Montag, 25. Januar 2010

Heap Dump Analysis with Memory Analyzer, Part 1: Heap Dumps

Almost two years passed since the Memory Analyzer tool (MAT) was published at Eclipse. Since then we have collected a lot of feedback, questions and comments by people using it, and we also gathered experience in using the tool ourselves. Most of the people find their way to solve memory problems using MAT relatively easy, but I am convinced there are also a lot of unexplored features and concepts within the tool, which can be very handy if properly understood and used. Therefore I decided to start a series of blog posts dedicated to memory analysis (with MAT) - starting from the basics and covering the different topics in detail. I would try to answer there some of the questions which pop-up most often, give some (hopefully useful) hints, explain the benefit of certain “unpopular” queries, and (please, please, please…) collect your feedback.

As the Memory Analyzer is a tool working with heap dumps, I will start with a detailed look at heap dumps – what they are, which formats MAT can read, what can be found inside, how one can get them, etc… If you are interested in the topic, read further.

What Is a Heap Dump?

A heap dump is a snapshot of the memory of a Java process at a certain point of time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the java objects and classes in the heap at the moment the snapshot was triggered. As it is just a snapshot at a given moment, a heap dump does not contain information such as when and where (in which method) an object was allocated.

What Are Heap Dumps Good for?

So what are heap dumps good for? Well, for a lot of things :-)
If there is a system which is crashing sporadically with an OutOfMemoryError, then analyzing an automatically written heap dump with MAT can be a very easy way to find the root cause of the problem (read more here).
If you wan to analyze what the footprint into memory of your application is, then MAT and heap dumps are again a good choice. This combination can also help you to find which are your biggest structures, to find redundant data structures, to find space wasted in unused collections, and much more. Such topics will be covered later in this blog series.
If you however are trying to find out why too many garbage objects are produced during a certain operation, or want to see which methods allocate most of the objects, then you would need to use a profiler which is collecting data over time from the VM. Leak detecting techniques relying on analysis of the objects behaviour (allocation / garbage collection) are difficult to inplement using heap dumps (see object identity below).

Types of Heap Dumps

Currently the Memory Analyzer is able to work with HPROF binary heap dumps (produced by Sun, HP, SAP, etc… JVMs), IBM system dumps (after preprocessing them), and IBM portable heap dumps (PHD) from a variaty of IBM platforms. Let’s have a closer look at each of the types.

HPROF Binary Heap Dumps

A detailed specification of the content of an HPROF file can be found here.

Below are summarized some of the important pieces of the information used within MAT:

  • Information about all loaded classes. For every class the HPROF dump contains its name, its super-class, its class loader, the defined fields for the instances (name and type), the static fields of the class and their values


  • Information about all objects. For every object one can find the class and the values of all fields – both references and primitive fields. The possibility to look at the names and the content of certain objects, e.g. the char[] within a huge StringBuilder, the size of a collection, etc … can be very helpful when performing memory analysis


  • a list of GC roots (what is a GC root?)


  • the callstacks of all threads (in heap dumps from JDK 6 update 14 and above)


  • IBM System Dumps

    On IBM platforms one can preprocess a system dump (core file) from a Java process with the jxtract tool, and analyze the result with Memory Analyzer on an arbitrary other box (DTFJ libraries have to be additionally installed, see details below in the “How to get heap dump” section). As the core file contains the whole process memory, this kind of dump also provides all the details seen in an hprof heap dump (including the field names, primitive fields' values, stacktraces, etc…). There is even more information (e.g. process related information), but at the moment it is not used in Memory Analyzer.

    IBM Portable Heap Dumps (PHD)

    The PHD files are much smaller in size than the corresponding system dumps. However, they contain less information.
    The major difference between the HPROF dumps (or the IBM system dumps) and PHD dumps is that a PHD dump does not contain the values of the primitive fields. Only the non-null references from an object are provided. The second important difference is that the field names are not present, i.e. one can’t distinguish from which field a reference is made, and because of this the presented reference chains (paths) are not as concrete as with the other dumps. Using just the object graph is still enough for the analysis of many memory-related problems, but when the content of some fields is needed to get an idea why an object is too big then one has to use the system dumps.
    Usually when a PHD dump is generated there is also a corresponding javacore file. If they are put together in the same directory when the PHD dump is opened with MAT, then some of the data in the javacore file will also be used.

    So, having less information has both advantages and disadvantages - the PHD dumps are ways easier to transport from a customer (smaller size), can be used to find the biggest objects in the heap. And as they are usually written by default they are a good place to start the analysis. However, in some cases the information is enough to analyze in details the root cause of a problem.

    A Common API for Them All?

    Having different formats for the heap dumps is definitely easier for the VM providers, as they can provide very efficiently the specific data they have. This however doesn’t hold true for the tools, which are faced with the different formats, have to understand each of them, and possibly optimize for every format separately.
    An attempt to solve this problem and make the life of tool writers easier is made under the Apache Kato project and the related JSR 326. They put efforts to provide a common API for accessing data from vendor specific snapshots and thus give tools a standard way to extract the data needed for post-mortem diagnostics (including memory related problems).

    How To Get a Heap Dump

    How to obtain a heap dump depends on the platform and the used JVM. In general all VMs provide the possibility to request a heap dump manually, or to get one written from the VM when an OutOfMemoryError occurs. The second option is very convenient for the analysis of problems happening on production systems, or happening only sporadically, as one does not have to observe the system and wait for the problem to reoccur.
    A detailed description how a heap dump can be obtained depending on the JVM is provided here.

    Object Identity

    One of the questions which we are asked very often is if MAT can recognize the same objects in two or more heap dumps from the same process. The answer is unfortunately still no. Object IDs which are provided in the known to us heap dumps are just the addresses at which the objects are located. As objects are often moved and reordered by the JVM during a GC these addressed change. Therefore they cannot be used to compare the objects. Tagging the objects while they are allocated is something a profiler could do (usually at a relatively high cost), but in the standard heap dumps described above such information is missing. Some ideas how to guess identical objects were discussed in this bugzilla entry.

    Are Dead Objects Present In the Heap Dump?

    Another question which often pops up is if garbage objects are included in the heap dump. This again depends on the heap dump, but usually a GC is done before the heap dump is written. Nevertheless there are always some objects which are unreachable from the GC roots, i.e. should be thrown away. The Memory Analyzer removes such objects during the initial parsing of a heap dump in order to simplify the analysis. If you want to have a look at the “garbage” or even want the objects to remain, then find here what to do.

    In Closing ...
    This was my attempt to give a detailed explanation of the different heap dump formats which the Memory Analyzer understands, and also give the answers to some of the questions which we frequently get. I'm sure there are still questions to be answered, and the MAT team will be very happy to get them from you, be it as comments here, in our fourm, or in bugzilla.

    51 Kommentare:

    1. the link for "detailed specification of the content of an HPROF file" https://heap-snapshot.dev.java.net/files/documents/4282/31543/hprof-binary-format.html is BROKEN

      AntwortenLöschen
      Antworten
      1. I replaced the link with a different one, which is working

        Löschen
    2. Nice post. I really enjoy reading it. Thanks for sharing.

      carpet cleaning

      AntwortenLöschen
    3. Nice post informative. One question as how to read on the current thread activity any possibilities via MAT?

      AntwortenLöschen
    4. Silk Road Research seeks to foster a reputation for integrity and professionalism. We are providing Strategic Growth Consultants, Market Research, Global Management Consulting, Investment Research and Business Growth, Research Consulting Firms/Companies in Singapore, USA, Asia and China.

      AntwortenLöschen
    5. Dieser Kommentar wurde vom Autor entfernt.

      AntwortenLöschen
    6. Có lẽ cần phải trải qua tuổi thanh xuân mới có thể hiểu được( cách dạy bé học toán lớp 4 tuổi ) tuổi xuân là khoảng thời gian ta( dạy trẻ học toán tư duy ) sống ích kỷ biết chừng nào. Có lúc nghĩ, sở dĩ tình yêu cần phải đi một vòng tròn lớn như vậy, phải trả một cái giá quá đắt như thế,( phương pháp dạy toán cho trẻ mầm non ) là bởi vì nó đến không đúng thời điểm. Khi có được( Toán mầm non ) tình yêu, chúng ta thiếu đi trí tuệ. Đợi đến khi( Bé học đếm số ) có đủ trí tuệ, chúng ta đã không còn sức lực để yêu một tình yêu thuần khiết nữa.

      AntwortenLöschen
    7. Có lẽ cần phải trải qua tuổi thanh xuân mới có thể hiểu được( cách dạy bé học toán lớp 4 tuổi ) tuổi xuân là khoảng thời gian ta( dạy trẻ học toán tư duy ) sống ích kỷ biết chừng nào. Có lúc nghĩ, sở dĩ tình yêu cần phải đi một vòng tròn lớn như vậy, phải trả một cái giá quá đắt như thế,( phương pháp dạy toán cho trẻ mầm non ) là bởi vì nó đến không đúng thời điểm. Khi có được( Toán mầm non ) tình yêu, chúng ta thiếu đi trí tuệ. Đợi đến khi( Bé học đếm số ) có đủ trí tuệ, chúng ta đã không còn sức lực để yêu một tình yêu thuần khiết nữa.

      AntwortenLöschen
    8. Thanks for sharing post.

      Hi, I am Sofi Hayat, I am working as a tech expert at email support. I have 3 years of experience in this field. If you have any problems related to Reset CenturyLink email password, etc, then please contact for instant help related to email problems.

      AntwortenLöschen
    9. Nice Post...
      Where can I register myself to access Microsoft applications?
      The only secured and well trusted place online to get an authorised Microsoft account is office.com. If you are a new user, you must know that to access any Microsoft application, users must have a product key. Just like any other paid software it ensure that you are using authentic software with unlimited services and no one has access to your account.
      office.com/setup
      www.office.com/setup

      AntwortenLöschen
    10. If you facing any issues due to your email get rid of all the emails related query. can connect our expert team engineer can resolve your all problems. | Fix ATT email login error

      AntwortenLöschen
    11. Nice content thanks for sharing with Us. I appreciate your knowledge I want to sharing some information about the Printers are you looking for Canon printer? and have any technical issues as Printer offline issues or error on the Operating System Windows 10, Mac iOS, We have highly experienced engineers can connect to resolve your printer errors online/offline.

      AntwortenLöschen
    12. I'm John Kelly, I belongs to United State, I work with "Officemyoffice" as a technical instructor, Are you looking for Microsoft Office install, download and re-activate visit the Office.com/setup and login your account and enter product key code (25 digit alphanumeric) and choose region or country then install/update.

      AntwortenLöschen
    13. The global   corn silk market is being aided by the growing production of corn. The global corn production is expected to attain a volume of 43731 million bushels between 2020 and 2021, while the global corn consumption is projected attain a volume of nearly 44690 million bushels during the same period. During 2021-2026, the global corn supply is projected to expand at a CAGR of 5.3%.

      AntwortenLöschen
    14. Rapid industrialization, especially in the developing economies, is one of the key factors driving the growth of the Antistatic Agents Market.

      AntwortenLöschen
    15. The global Automotive Fender Roller Marketto be aided by the growth of the automotive garage equipment market, which reached a value of about USD 7.3 billion in 2020. The automotive garage equipment market is further expected to grow at a CAGR of 6% in the forecast period of 2021-2026 to reach a value of approximately USD 9.26 billion by 2026.

      AntwortenLöschen
    16.  Benzyl Bromide Market is used in the synthesis of a wide variety of organic compounds, leading to applications in numerous industries. The steady increase in the use of chemicals in industries, as well as the growth of industries such as pharmaceuticals, pesticides, and paints in which benzyl bromide has end uses, is expected to lead to the healthy growth of the benzyl bromide industry during the forecast period (2021-2026).

      AntwortenLöschen
    17. I am working as an SMM Panel. I have 3 years of experience in this field. If you have any problems related to the SMM panel, etc, then please contact me for instant help related to the SMM panel.

      AntwortenLöschen
    18. The easiest way to get more views on your Instagram page. Buy Instagram views from SMM panel. Post consistently on your social media page or create your profile more attractive. These steps help you for growth your page organically. For more information You can connect with our website.
      Buy Instagram views

      AntwortenLöschen
    19. The global Digital Health Market size was valued at USD 96.5 billion in 2020 and is expected to grow at a compound annual growth rate (CAGR) of 15.1% from 2021 to 2028. Increasing cases of obesity, high prevalence of chronic diseases, such as diabetes, Cardiovascular Disorders (CVDs), growing need for remote patient monitoring services, and increasing patient engagement & connectivity owing to high adoption of digital health technology are the major factors promoting the market growth. According to research data published in NCBI in 2018, 95.0% of the U.S. adult population possessed a mobile phone, with 64.0% of them owning smartphones. As per the report published by the GSM Association, Mobile Economy 2018, the number of people connected to mobile services surpassed 5 billion in 2017 and the number of unique mobile subscribers is anticipated to reach 5.9 billion by 2025.

      AntwortenLöschen

    20. Aluminium Fluoride Market
      size is likely to witness significant growth in the coming years on account of rapid proliferation of processed beverage & food products, transport industry growth and technological advancements. Healthy economic growth combined with the increasing presence of female workforce should result in rising preference for convenience packaged beverage and food items. Aluminum fluoride is widely used to make aluminium cans which are gaining popularity owing to their rapid recycling rates around the globe, which is likely to boost aluminum fluoride market growth.

      AntwortenLöschen
    21. The Global Pet Tech Market is segmented based on product type, tracking equipment, monitoring equipment, entertainment equipment, feeding equipment, etc. Tracking equipment evolved as the largest segment in the global pet tech market owing to growing investment by the pet owners in the active tracking of health and behavioral aspects of their pets.

      AntwortenLöschen
    22. The Germany Dairy Market is being driven by the rising health consciousness among the consumers, thus, leading to a rising demand for healthier dairy products like unprocessed cheese. Soft cheese and other protein-rich dairy products are also witnessing a growth in the country due to the decreasing meat consumption in the region.

      AntwortenLöschen
    23. The Automotive Seats Market in the pre-COVID-19 situation was projected to reach USD 73.0 billion by 2025 from USD 64.3 million in 2020. However, due to the COVID-19 pandemic, OEMs are likely to reduce financial investments in R&D activities related to automotive seats. Also, due to lockdowns, manufacturing units around the world were shut down, as a result of which vehicle production and sales have taken a huge hit.

      AntwortenLöschen
    24. The global Fleet Management Market size is expected to grow from USD 19.9 billion in 2020 to USD 34.0 billion by 2025, at a Compound Annual Growth Rate (CAGR) of 11.3% during the forecast period. The growth of the market can be attributed to the cost benefits versus Software-as-a-Service (SaaS) and cloud-based deployments of fleet management solutions. The primary factors driving the growth of the fleet management market include the increasing government regulations and need for optimization of fleet operating expenses. The other factors supporting the market growth include the decreasing hardware and software costs and growing need for operational efficiency among fleet owners.

      AntwortenLöschen
    25. Thanks for sharing this informative post. I recommend keep sharing like this. I also recommend to visit ABC assignment help website for amazing blogs and assignment writing service in Australia.
      assignment help

      AntwortenLöschen
    26. Thank you for some other informative blog. Where else could I get that type of information written in such an ideal means? I have a mission that I’m just now working on, and I have been at the look out for such information. Also visit my website: 카지노사이트

      AntwortenLöschen
    27. Its an amazing website, really enjoy your articles. Helpful and interesting too. Keep doing this in future. I will support you.Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. Also visit my website: 카지노사이트

      AntwortenLöschen
    28. great submit, very informative. I’m wondering why the other experts of this sector do not notice this. You must continue your writing. I am confident, you have a great readers’ base already! Also visit my website: 카지노사이트

      AntwortenLöschen
    29. magnificent put up, very informative. I'm wondering why the opposite experts of this sector don't notice this. You must continue your writing. I am confident, you have a huge readers' base already! Also visit my website: 카지노사이트

      AntwortenLöschen
    30. You will never get the good assignments from a bad company. As we all know that getting an assignment help service is very important to take at this era and searching for that is almost impossible task. People used to find on the basis of the reviews, but GoAssignmentHelp has created a hype in this era by proving themselves as best assignment help company in Australia which is covering the whole globe for providing the assignment help services with the tools which help in providing 100% unique assignments. You can check out for the best plagiarism checker services provided by us.

      AntwortenLöschen
    31. Researching on a subject and writing a plagiarism-free assignment thereon are often quite nerve-wracking task for a student. So, irrespective of how weird or tough your assignment topic is you'll be able to forever have faith in Assignment Help to supply you with the simplest assignment.

      AntwortenLöschen
    32. Love to read all post as it encourages me to scratch more and more to make my study objective meaningful. Continue such amazing as I am big fan of thought expression style. Let us get in touch with the navigational link of assignment help to see what is going new.

      AntwortenLöschen
    33. Enjoyed reading the article above, really explains everything in detail,the article is very interesting and effective. Thank you and good luck for the upcoming articles credit for this ans yourhomes

      AntwortenLöschen
    34. Hello, I enjoy reading all of your post. I like to write a little comment to support you.Click Me Here 경마사이트



      AntwortenLöschen
    35. McAfee is the best and most trusted MCafee not working program that has been crafted with impressive features or ability to identify malicious viruses that are on your laptop, computer or other Android device. When you Mcafee password Reset your program, you will be able to benefit from its advanced technology and other features. If you're having problems while trying to updating McAfee Antivirus on Windows 10. Feel free to contact us at the Antivirus Support Team. You can also use live chat or email us for us for a quick resolution.
      Our expert team is all-hours available to help you. HP 4500 Printer Offline

      AntwortenLöschen
    36. Very impressive post i appreciate you. thanks for sharing with us. If you are looking for all types of details regarding. SGPGI PDCC 2022 Then you must have a look at our website. You will get all the details including exam preparation, syllabus, exam date-sheet, cut-off, and other things in detail. For more information, visit our website.

      AntwortenLöschen
    37. Thanks for the fantastic post, very informative I wonder why other experts in this area have not seen this. You must continue your writing. Many thanks so much for sharing. Also visit free nnpc aptitude test past questions & solutions pdf format

      AntwortenLöschen
    38. I read your post and I liked your post very much. Keep posting like this. And also visit. Assignment Help

      AntwortenLöschen
    39. Dieser Kommentar wurde vom Autor entfernt.

      AntwortenLöschen
    40. Thanks a LOT! the way you explained head dumps is so accurate. accounting assignment help , expert acounts paper writing solution for both students and professionals. Its
      -Budget-freindly
      -Expert written and insightful
      -Customizable and plagiarism and AI free content.

      AntwortenLöschen
    41. Please continue sharing such informative blogs; they are genuinely helpful. Looking forward to more content from you in the future.
      modern dog accessories

      AntwortenLöschen
    42. Management assignment help is an assignment writer, and they deliver their assignment services worldwide. Global assignment experts are known for their assignment writing and for delivering assignments on time. Global assignment help has a team of experienced experts and they will help you with all your subjects and clear your doubts related with your subjects. For all your assignment solutions, contact with us on our website and we will help you with your assignment.

      AntwortenLöschen
    43. I'm really enjoying the content on this blog. It's informative and well-written. Looking forward to more posts like this. Keep up the great work...food dispensing dog toy

      AntwortenLöschen