{"id":3455,"date":"2025-04-22T12:46:31","date_gmt":"2025-04-22T12:46:31","guid":{"rendered":"https:\/\/kedar.nitty-witty.com\/blog\/?p=3455"},"modified":"2025-04-22T12:58:54","modified_gmt":"2025-04-22T12:58:54","slug":"how-to-resolve-disk-space-issues-in-pmm-case-study","status":"publish","type":"post","link":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study","title":{"rendered":"How to Resolve Disk Space Issues in PMM: Case Study"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Recently, I encountered a Percona Monitoring and Management (PMM) server that was rapidly approaching complete disk exhaustion. This post outlines the steps taken to identify the issue and reclaim disk space.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ df -h\nFilesystem Size Used Avail Use% Mounted on\n\/dev\/sda1 523G 508G 16G 98% \/<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With only 16GB remaining on a 523GB drive, this PMM installation was on the brink of failure. If you&#8217;re experiencing similar issues with your PMM deployment, this guide details how to identify and resolve PMM disk space issues.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">BTW did you know, <a href=\"https:\/\/docs.percona.com\/percona-monitoring-and-management\/3\/release-notes\/3.0.0.html\" target=\"_blank\" rel=\"noopener nofollow\" title=\"\">PMM 3.0<\/a> is already out?<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">Identifying the Culprit<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">After inspecting common culprits like system logs and temporary files, the main disk usage was traced to <code>\/var\/local\/percona\/pmm\/srv\/<\/code>, a mount used by the PMM container. Within it, the ClickHouse database specific folder was the primary space consumer.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">ClickHouse in PMM<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Probably you already know but if not, question is, what&#8217;s ClickHouse is housing in PMM&#8217;s architecture that is utilizing so much disk? <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ClickHouse is an open-source, column-oriented database management system that excels at real-time analytics on large datasets. Within the PMM ecosystem, <strong>ClickHouse<\/strong> facilitates the <strong>Query Analytics<\/strong> functionality. This explains why the ClickHouse database can grow significantly over time, especially in environments monitoring numerous database instances with heavy query loads.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Investigation: Tracking Down the Biggest Tables<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To identify which specific tables were responsible for the excessive disk usage, I connected to the PMM server container:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Connect to the PMM server container\npodman exec -it pmm-server \/bin\/bash\n\n# or if you're using docker\ndocker exec -it pmm-server \/bin\/bash<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Once inside the container, following queries was run against ClickHouse to identify the largest tables:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ clickhouse-client<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT database, table, SUM(bytes_on_disk) AS size_on_disk, COUNT() AS parts_count \nFROM system.parts \nWHERE active = 1 \nGROUP BY database, table \nORDER BY size_on_disk DESC;<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>\u250c\u2500database\u2500\u252c\u2500table\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500size_on_disk\u2500\u252c\u2500parts_count\u2500\u2510\n\u2502 system   \u2502 trace_log                 \u2502 346710891455 \u2502          48 \u2502\n\u2502 system   \u2502 asynchronous_metric_log   \u2502   4216804400 \u2502          37 \u2502\n\u2502 system   \u2502 metric_log_0              \u2502   4096890831 \u2502          84 \u2502\n\u2502 system   \u2502 metric_log                \u2502   3300187902 \u2502          35 \u2502\n...<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>system.trace_log<\/code> table was consuming a whopping 346GB! This table stores trace information for ClickHouse queries and operations.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Drilling Down: Analyzing Partitions<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To understand the issue with the trace_log table, its partition details were reviewed:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT database, table, partition, sum(rows) AS rows, sum(bytes_on_disk) AS size_on_disk \nFROM system.parts \nWHERE active = 1 AND table = 'trace_log' \nGROUP BY database, table, partition \nORDER BY partition ASC;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The output of above query exhibited the problem:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\u250c\u2500database\u2500\u252c\u2500table\u2500\u2500\u2500\u2500\u2500\u252c\u2500partition\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500rows\u2500\u252c\u2500size_on_disk\u2500\u2510\n\u2502 system   \u2502 trace_log \u2502 202405    \u2502   19230736 \u2502    353844712 \u2502\n\u2502 system   \u2502 trace_log \u2502 202406    \u2502 2972432060 \u2502  48032655534 \u2502\n\u2502 system   \u2502 trace_log \u2502 202407    \u2502 3500442984 \u2502  56447376224 \u2502\n\u2502 system   \u2502 trace_log \u2502 202408    \u2502 3381851250 \u2502  53964707527 \u2502\n\u2502 system   \u2502 trace_log \u2502 202409    \u2502 3138145807 \u2502  50178088805 \u2502\n\u2502 system   \u2502 trace_log \u2502 202410    \u2502 3307448245 \u2502  52953410205 \u2502\n\u2502 system   \u2502 trace_log \u2502 202411    \u2502 3184102845 \u2502  51206343208 \u2502\n\u2502 system   \u2502 trace_log \u2502 202412    \u2502 2093835550 \u2502  33552956631 \u2502\n\u2502 system   \u2502 trace_log \u2502 202501    \u2502    1183332 \u2502     21518446 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The table contained partitions dating back over 8 months! This historical data was consuming massive amounts of disk space but provided little value for current monitoring needs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Looking at the Table Structure<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>SHOW CREATE TABLE system.trace_log;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The output confirmed that the table was partitioned by month but had no TTL (Time To Live) policy to automatically purge old data:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CREATE TABLE system.trace_log\n(\n    `event_date` Date,\n    `event_time` DateTime,\n    ...\n)\nENGINE = MergeTree\nPARTITION BY toYYYYMM(event_date)\nORDER BY (event_date, event_time)\nSETTINGS index_granularity = 8192<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">The Solution: Two-Part Approach<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. Implementing Automatic Data Retention with TTL<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Our first step was to add a TTL policy to automatically remove data older than 30 days:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Edit the ClickHouse configuration\nvi \/etc\/clickhouse-server\/config.xml<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">added the following TTL definition in the specific section for the table trace_log:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;ttl&gt;event_date + INTERVAL 30 DAY DELETE&lt;\/ttl&gt;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then restarted ClickHouse to apply the changes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>supervisorctl restart clickhouse<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After the restart, we verified the change was applied by checking the table definition again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>SHOW CREATE TABLE system.trace_log;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The updated definition now included:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@pmm-host-server opt]# clickhouse-client\nClickHouse client version 23.8.2.7 (official build).\nConnecting to localhost:9000 as user default.\nConnected to ClickHouse server version 23.8.2 revision 54465.\n\npmm-host-server :) SHOW CREATE TABLE system.trace_log;\n\nSHOW CREATE TABLE system.trace_log\n\nQuery id: 4a2f87cb-0030-4751-b17d-08aa84f38abb\n\n\u250c\u2500statement\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 CREATE TABLE system.trace_log\n(\n    `event_date` Date,\n    `event_time` DateTime,\n    `event_time_microseconds` DateTime64(6),\n    `timestamp_ns` UInt64,\n    `revision` UInt32,\n    `trace_type` Enum8('Real' = 0, 'CPU' = 1, 'Memory' = 2, 'MemorySample' = 3, 'MemoryPeak' = 4, 'ProfileEvent' = 5),\n    `thread_id` UInt64,\n    `query_id` String,\n    `trace` Array(UInt64),\n    `size` Int64,\n    `ptr` UInt64,\n    `event` LowCardinality(String),\n    `increment` Int64\n)\nENGINE = MergeTree\nPARTITION BY toYYYYMM(event_date)\nORDER BY (event_date, event_time)\n<strong>TTL event_date + toIntervalDay(30)<\/strong>\nSETTINGS index_granularity = 8192 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">2. Manually Removing Historical Data<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">I decided to manually remove the existing historical data to reclaim disk space immediately and began dropping old partitions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ALTER TABLE system.trace_log_1 DROP PARTITION '202405';\nALTER TABLE system.trace_log_1 DROP PARTITION '202406';<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">But it wasn&#8217;t straight forward\u2026 I hit a roadblock by configuration variable<em> max_[table\/partition]_size_to_drop<\/em> as the partition being dropped is more than 50G the operation was aborted with an error and possible solutions.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pmm-host-server :) ALTER TABLE system.trace_log_1 DROP PARTITION '202407';\n\nALTER TABLE system.trace_log_1\n    DROP PARTITION '202407'\n\nQuery id: f2b93063-57df-444f-9199-82c7695f39b9\n\n\n0 rows in set. Elapsed: 0.003 sec.\n\nReceived exception from server (version 23.8.2):\nCode: 359. DB::Exception: Received from localhost:9000. DB::Exception: Table or Partition in system.trace_log_1 was not dropped.\nReason:\n1. Size (56.45 GB) is greater than max_&#91;table\/partition]_size_to_drop (50.00 GB)\n2. File '\/srv\/clickhouse\/flags\/force_drop_table' intended to force DROP doesn't exist\nHow to fix this:\n1. Either increase (or set to zero) max_&#91;table\/partition]_size_to_drop in server config\n2. Either create forcing file \/srv\/clickhouse\/flags\/force_drop_table and make sure that ClickHouse has write permission for it.\nExample:\nsudo touch '\/srv\/clickhouse\/flags\/force_drop_table' &amp;&amp; sudo chmod 666 '\/srv\/clickhouse\/flags\/force_drop_table'. (TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT)\n\npmm-host-server :)\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>ClickHouse has a safety mechanism that prevents dropping large partitions (&gt;50GB) without explicit confirmation.<\/strong> <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To work around this, I created a force drop flag as noted in the instructions above:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>touch '\/srv\/clickhouse\/flags\/force_drop_table' &amp;&amp; chmod 666 '\/srv\/clickhouse\/flags\/force_drop_table'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With this flag in place, it&#8217;s possible to drop the larger partitions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ALTER TABLE system.trace_log_1 DROP PARTITION '202407';<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Note that the flag is automatically removed after each operation, so it needed to be recreated before dropping each large partition.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Results and Long-term Prevention<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">After implementing our solution, I successfully reduced the disk usage and established a sustainable automatic cleanup process. The TTL configuration ensures that trace logs older than 30 days will be automatically purged, preventing future disk space issues.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The Percona team is aware of this issue and is working on implementing better default TTL policies for system tables in future PMM releases. A bug report (<a href=\"https:\/\/perconadev.atlassian.net\/browse\/PMM-13644\" target=\"_blank\" rel=\"noopener nofollow\" title=\"\">PMM-13644<\/a>) has been filed to address this issue more permanently.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Until these improvements are implemented in future PMM versions, the workaround described in this case study will help you manage ClickHouse disk usage effectively.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Reclaiming disk space in PMM is crucial for maintaining optimal server performance and monitoring efficiency. As shown in this case study, understanding PMM&#8217;s internal components, such as the <code>trace_log<\/code> table in ClickHouse, helps identify the root cause of disk usage issues.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Have you faced similar issues with your PMM deployment? Share your strategies for managing disk usage in the comments below!<\/p>\n","protected":false},"excerpt":{"rendered":"Recently, I encountered a Percona Monitoring and Management (PMM) server that was rapidly approaching complete disk exhaustion. This post outlines the steps taken to identify the issue and reclaim disk&hellip;\n","protected":false},"author":1,"featured_media":3462,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[8,1089],"tags":[1079,1077,427,1085,1084,1082,1087,1088,1080],"class_list":["post-3455","post","type-post","status-publish","format-standard","has-post-thumbnail","category-mysql","category-percona-monitoring-and-management","tag-clickhouse","tag-disk-space","tag-mysql","tag-pmm","tag-pmm-case-study","tag-pmm-server","tag-reclaim-disk-space","tag-resource-management","tag-system-stability"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Kedar\"\/>\n\t<meta name=\"google-site-verification\" content=\"_Bu1h5r0PxIC0D2jBVdmYN45RTeG9ogcX85XLkbG1qA\" \/>\n\t<meta name=\"msvalidate.01\" content=\"116B62074CBCA1BA99B8CCE09CCF2FB8\" \/>\n\t<meta name=\"keywords\" content=\"clickhouse,disk space,mysql,pmm,pmm case study,pmm server,reclaim disk space,resource management,system stability\" \/>\n\t<link rel=\"canonical\" href=\"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Change Is Inevitable | Kedar Vaijanapurkar&#039;s Blog for MySQL, technology and various subjects\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable\" \/>\n\t\t<meta property=\"og:description\" content=\"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2025-04-22T12:46:31+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2025-04-22T12:58:54+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary\" \/>\n\t\t<meta name=\"twitter:title\" content=\"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable\" \/>\n\t\t<meta name=\"twitter:description\" content=\"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#article\",\"name\":\"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable\",\"headline\":\"How to Resolve Disk Space Issues in PMM: Case Study\",\"author\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/author\\\/admin#author\"},\"publisher\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/#person\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/1745326202527-1.avif\",\"width\":800,\"height\":533,\"caption\":\"pmm-architecture-ghibli-style\"},\"datePublished\":\"2025-04-22T12:46:31+00:00\",\"dateModified\":\"2025-04-22T12:58:54+00:00\",\"inLanguage\":\"en-US\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#webpage\"},\"isPartOf\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#webpage\"},\"articleSection\":\"MySQL, Percona Monitoring and Management, ClickHouse, disk space, MySQL, PMM, PMM case study, PMM server, reclaim disk space, resource management, system stability\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql#listItem\",\"name\":\"MySQL\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql#listItem\",\"position\":2,\"name\":\"MySQL\",\"item\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql\\\/percona-monitoring-and-management#listItem\",\"name\":\"Percona Monitoring and Management\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql\\\/percona-monitoring-and-management#listItem\",\"position\":3,\"name\":\"Percona Monitoring and Management\",\"item\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql\\\/percona-monitoring-and-management\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#listItem\",\"name\":\"How to Resolve Disk Space Issues in PMM: Case Study\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql#listItem\",\"name\":\"MySQL\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#listItem\",\"position\":4,\"name\":\"How to Resolve Disk Space Issues in PMM: Case Study\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/category\\\/mysql\\\/percona-monitoring-and-management#listItem\",\"name\":\"Percona Monitoring and Management\"}}]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/#person\",\"name\":\"Kedar\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#personImage\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/be1f7021485c325d92cc4c5d961accb9c1af72b1b11281a790f7f4f066ef10d3?s=96&d=mm&r=g\",\"width\":96,\"height\":96,\"caption\":\"Kedar\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/author\\\/admin#author\",\"url\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/author\\\/admin\",\"name\":\"Kedar\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#authorImage\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/be1f7021485c325d92cc4c5d961accb9c1af72b1b11281a790f7f4f066ef10d3?s=96&d=mm&r=g\",\"width\":96,\"height\":96,\"caption\":\"Kedar\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#webpage\",\"url\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study\",\"name\":\"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable\",\"description\":\"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study\",\"inLanguage\":\"en-US\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/#website\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#breadcrumblist\"},\"author\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/author\\\/admin#author\"},\"creator\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/author\\\/admin#author\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/1745326202527-1.avif\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study\\\/#mainImage\",\"width\":800,\"height\":533,\"caption\":\"pmm-architecture-ghibli-style\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/how-to-resolve-disk-space-issues-in-pmm-case-study#mainImage\"},\"datePublished\":\"2025-04-22T12:46:31+00:00\",\"dateModified\":\"2025-04-22T12:58:54+00:00\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/\",\"name\":\"..::CHANGE is INEVITABLE::..\",\"description\":\"Kedar Vaijanapurkar's Blog for MySQL, technology and various subjects\",\"inLanguage\":\"en-US\",\"publisher\":{\"@id\":\"https:\\\/\\\/kedar.nitty-witty.com\\\/blog\\\/#person\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable","description":"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study","canonical_url":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study","robots":"max-image-preview:large","keywords":"clickhouse,disk space,mysql,pmm,pmm case study,pmm server,reclaim disk space,resource management,system stability","webmasterTools":{"google-site-verification":"_Bu1h5r0PxIC0D2jBVdmYN45RTeG9ogcX85XLkbG1qA","msvalidate.01":"116B62074CBCA1BA99B8CCE09CCF2FB8","miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#article","name":"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable","headline":"How to Resolve Disk Space Issues in PMM: Case Study","author":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/author\/admin#author"},"publisher":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/#person"},"image":{"@type":"ImageObject","url":"https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2025\/04\/1745326202527-1.avif","width":800,"height":533,"caption":"pmm-architecture-ghibli-style"},"datePublished":"2025-04-22T12:46:31+00:00","dateModified":"2025-04-22T12:58:54+00:00","inLanguage":"en-US","mainEntityOfPage":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#webpage"},"isPartOf":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#webpage"},"articleSection":"MySQL, Percona Monitoring and Management, ClickHouse, disk space, MySQL, PMM, PMM case study, PMM server, reclaim disk space, resource management, system stability"},{"@type":"BreadcrumbList","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog#listItem","position":1,"name":"Home","item":"https:\/\/kedar.nitty-witty.com\/blog","nextItem":{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql#listItem","name":"MySQL"}},{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql#listItem","position":2,"name":"MySQL","item":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql","nextItem":{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\/percona-monitoring-and-management#listItem","name":"Percona Monitoring and Management"},"previousItem":{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog#listItem","name":"Home"}},{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\/percona-monitoring-and-management#listItem","position":3,"name":"Percona Monitoring and Management","item":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\/percona-monitoring-and-management","nextItem":{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#listItem","name":"How to Resolve Disk Space Issues in PMM: Case Study"},"previousItem":{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql#listItem","name":"MySQL"}},{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#listItem","position":4,"name":"How to Resolve Disk Space Issues in PMM: Case Study","previousItem":{"@type":"ListItem","@id":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\/percona-monitoring-and-management#listItem","name":"Percona Monitoring and Management"}}]},{"@type":"Person","@id":"https:\/\/kedar.nitty-witty.com\/blog\/#person","name":"Kedar","image":{"@type":"ImageObject","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#personImage","url":"https:\/\/secure.gravatar.com\/avatar\/be1f7021485c325d92cc4c5d961accb9c1af72b1b11281a790f7f4f066ef10d3?s=96&d=mm&r=g","width":96,"height":96,"caption":"Kedar"}},{"@type":"Person","@id":"https:\/\/kedar.nitty-witty.com\/blog\/author\/admin#author","url":"https:\/\/kedar.nitty-witty.com\/blog\/author\/admin","name":"Kedar","image":{"@type":"ImageObject","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#authorImage","url":"https:\/\/secure.gravatar.com\/avatar\/be1f7021485c325d92cc4c5d961accb9c1af72b1b11281a790f7f4f066ef10d3?s=96&d=mm&r=g","width":96,"height":96,"caption":"Kedar"}},{"@type":"WebPage","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#webpage","url":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study","name":"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable","description":"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study","inLanguage":"en-US","isPartOf":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/#website"},"breadcrumb":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#breadcrumblist"},"author":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/author\/admin#author"},"creator":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/author\/admin#author"},"image":{"@type":"ImageObject","url":"https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2025\/04\/1745326202527-1.avif","@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study\/#mainImage","width":800,"height":533,"caption":"pmm-architecture-ghibli-style"},"primaryImageOfPage":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study#mainImage"},"datePublished":"2025-04-22T12:46:31+00:00","dateModified":"2025-04-22T12:58:54+00:00"},{"@type":"WebSite","@id":"https:\/\/kedar.nitty-witty.com\/blog\/#website","url":"https:\/\/kedar.nitty-witty.com\/blog\/","name":"..::CHANGE is INEVITABLE::..","description":"Kedar Vaijanapurkar's Blog for MySQL, technology and various subjects","inLanguage":"en-US","publisher":{"@id":"https:\/\/kedar.nitty-witty.com\/blog\/#person"}}]},"og:locale":"en_US","og:site_name":"Change Is Inevitable | Kedar Vaijanapurkar's Blog for MySQL, technology and various subjects","og:type":"article","og:title":"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable","og:description":"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study","og:url":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study","article:published_time":"2025-04-22T12:46:31+00:00","article:modified_time":"2025-04-22T12:58:54+00:00","twitter:card":"summary","twitter:title":"How to Resolve Disk Space Issues in PMM: Case Study | Change Is Inevitable","twitter:description":"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study"},"aioseo_meta_data":{"post_id":"3455","title":null,"description":"How to reclaim disk space in PMM and optimize server performance with practical strategies like TTL policies and data cleanup, backed by a real-world case study","keywords":null,"keyphrases":{"focus":{"keyphrase":"","score":0,"analysis":{"keyphraseInTitle":{"score":0,"maxScore":9,"error":1}}},"additional":[]},"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":"","og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"Article","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":"-1","robots_max_videopreview":"-1","robots_max_imagepreview":"large","priority":null,"frequency":"default","location":null,"local_seo":null,"breadcrumb_settings":null,"limit_modified_date":false,"ai":null,"created":"2025-04-22 10:53:38","updated":"2025-08-16 19:23:36","seo_analyzer_scan_date":null},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/kedar.nitty-witty.com\/blog\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">\u00bb<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\" title=\"MySQL\">MySQL<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">\u00bb<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\/percona-monitoring-and-management\" title=\"Percona Monitoring and Management\">Percona Monitoring and Management<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">\u00bb<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tHow to Resolve Disk Space Issues in PMM: Case Study\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/kedar.nitty-witty.com\/blog"},{"label":"MySQL","link":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql"},{"label":"Percona Monitoring and Management","link":"https:\/\/kedar.nitty-witty.com\/blog\/category\/mysql\/percona-monitoring-and-management"},{"label":"How to Resolve Disk Space Issues in PMM: Case Study","link":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-resolve-disk-space-issues-in-pmm-case-study"}],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts\/3455","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/comments?post=3455"}],"version-history":[{"count":3,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts\/3455\/revisions"}],"predecessor-version":[{"id":3460,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts\/3455\/revisions\/3460"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/media\/3462"}],"wp:attachment":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/media?parent=3455"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/categories?post=3455"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/tags?post=3455"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}