----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Using the Valgrind API in mysqld CREATION DATE..: Fri, 22 May 2009, 11:43 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 23 (http://askmonty.org/worklog/?tid=23) VERSION........: Server-5.1 STATUS.........: Code-Review PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 40 (hours remain) ORIG. ESTIMATE.: 40 PROGRESS NOTES: -=-=(Knielsen - Mon, 10 Aug 2009, 14:27)=-=- Low Level Design modified. --- /tmp/wklog.23.old.16018 2009-08-10 14:27:09.000000000 +0300 +++ /tmp/wklog.23.new.16018 2009-08-10 14:27:09.000000000 +0300 @@ -5,3 +5,5 @@ - sql/item_strfunc.cc (Item_func_compress). +Another good place is in the TRASH_MEM macro. + -=-=(Knielsen - Wed, 24 Jun 2009, 15:55)=-=- Supervisor updated. --- /tmp/wklog.23.old.944 2009-06-24 15:55:57.000000000 +0300 +++ /tmp/wklog.23.new.944 2009-06-24 15:55:57.000000000 +0300 @@ -1 +1 @@ -Knielsen +Monty -=-=(Knielsen - Wed, 24 Jun 2009, 15:53)=-=- Version updated. --- /tmp/wklog.23.old.911 2009-06-24 15:53:32.000000000 +0300 +++ /tmp/wklog.23.new.911 2009-06-24 15:53:32.000000000 +0300 @@ -1 +1 @@ -Maria-1.0 +Server-5.1 -=-=(Knielsen - Wed, 24 Jun 2009, 15:52)=-=- Version updated. --- /tmp/wklog.23.old.897 2009-06-24 15:52:43.000000000 +0300 +++ /tmp/wklog.23.new.897 2009-06-24 15:52:43.000000000 +0300 @@ -1 +1 @@ -Connector/.NET-2.1 +Maria-1.0 -=-=(Knielsen - Wed, 24 Jun 2009, 15:52)=-=- Version updated. --- /tmp/wklog.23.old.895 2009-06-24 15:52:28.000000000 +0300 +++ /tmp/wklog.23.new.895 2009-06-24 15:52:28.000000000 +0300 @@ -1 +1 @@ -Maria-1.0 +Connector/.NET-2.1 -=-=(Knielsen - Wed, 24 Jun 2009, 15:35)=-=- Version updated. --- /tmp/wklog.23.old.32742 2009-06-24 15:35:48.000000000 +0300 +++ /tmp/wklog.23.new.32742 2009-06-24 15:35:48.000000000 +0300 @@ -1 +1 @@ -Server-5.1 +Maria-1.0 -=-=(Knielsen - Fri, 22 May 2009, 14:31)=-=- Low Level Design modified. --- /tmp/wklog.23.old.24587 2009-05-22 14:31:52.000000000 +0300 +++ /tmp/wklog.23.new.24587 2009-05-22 14:31:52.000000000 +0300 @@ -1 +1,7 @@ +Two places where we call into libz, and where checking for defined parameters +would be good: + + - mysys/my_compress.c + + - sql/item_strfunc.cc (Item_func_compress). -=-=(Guest - Fri, 22 May 2009, 12:04)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.18061 2009-05-22 12:04:05.000000000 +0300 +++ /tmp/wklog.23.new.18061 2009-05-22 12:04:05.000000000 +0300 @@ -26,3 +26,5 @@ initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. + +See also a patch here: http://bugs.mysql.com/bug.php?id=44582 -=-=(Knielsen - Fri, 22 May 2009, 11:52)=-=- High-Level Specification modified. --- /tmp/wklog.23.old.17628 2009-05-22 11:52:33.000000000 +0300 +++ /tmp/wklog.23.new.17628 2009-05-22 11:52:33.000000000 +0300 @@ -1 +1,28 @@ +With custom memory allocators, using the Valgrind APIs we can tell Valgrind when +a memory block is allocated (so that data read from memory is marked as undefined +instead of being defined or not at random depending on prior use); and when a +memory block is freed (so that use after freeing can be reported as an error). +In some cases cheking for leaks may also be appropriate. + +Another possibility is to add an explicit check for whether memory is defined. + +One place this would be useful is when calling libz. Due to the design of that +library, Valgrind produces lots of false alarms about using undefined values +(I think the issue is that it runs a few bytes off of initialized memory to +reduce boundary checks in each loop iteration, then after the loop has checks to +avoid using the undefined part of the result). This means we have lots of libz +Valgrind suppressions and continue to add more as new warnings surface. So we +might easily miss a real problem in this area. This could be improved by adding +explicit checks at the call to libz functions that the passed memory is properly +defined. + +Another use is to improve debugging. It is often the case when debugging a +warning about using un-initialised memory that the detection happens long after +the real problem, the un-initialized value being passed along through the code +for a long time before being detected. This makes debugging the problem slow. + +By adding in strategic places code that asserts that a specific value must be +initialised, it is possible to detect problems earlier, speeding up debugging. +Such code can be added in more places over time as development and debugging +goes on. DESCRIPTION: Valgrind (the memcheck tool) has some very useful APIs that can be used in mysqld when testing with Valgrind to improve testing and/or debugging: file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.clientreqs file:///usr/share/doc/valgrind/html/mc-manual.html#mc-manual.mempools This worklog is about adding configure checks and headers to allow to use these in a way that continues to work on machines where the Valgrind headers or functionality is missing. It also includes adding some basic Valgrind enhancements: - Adding Valgrind annotations to custom memory allocators so that Valgrind can detect leaks, use-before-init, and use-after-free problems also for these allocators. - Adding checks for definedness in appropriate places (eg. when calling libz). HIGH-LEVEL SPECIFICATION: With custom memory allocators, using the Valgrind APIs we can tell Valgrind when a memory block is allocated (so that data read from memory is marked as undefined instead of being defined or not at random depending on prior use); and when a memory block is freed (so that use after freeing can be reported as an error). In some cases cheking for leaks may also be appropriate. Another possibility is to add an explicit check for whether memory is defined. One place this would be useful is when calling libz. Due to the design of that library, Valgrind produces lots of false alarms about using undefined values (I think the issue is that it runs a few bytes off of initialized memory to reduce boundary checks in each loop iteration, then after the loop has checks to avoid using the undefined part of the result). This means we have lots of libz Valgrind suppressions and continue to add more as new warnings surface. So we might easily miss a real problem in this area. This could be improved by adding explicit checks at the call to libz functions that the passed memory is properly defined. Another use is to improve debugging. It is often the case when debugging a warning about using un-initialised memory that the detection happens long after the real problem, the un-initialized value being passed along through the code for a long time before being detected. This makes debugging the problem slow. By adding in strategic places code that asserts that a specific value must be initialised, it is possible to detect problems earlier, speeding up debugging. Such code can be added in more places over time as development and debugging goes on. See also a patch here: http://bugs.mysql.com/bug.php?id=44582 LOW-LEVEL DESIGN: Two places where we call into libz, and where checking for defined parameters would be good: - mysys/my_compress.c - sql/item_strfunc.cc (Item_func_compress). Another good place is in the TRASH_MEM macro. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)