{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5182","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5182/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5182/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5182/events","html_url":"https://github.com/zfsonlinux/zfs/pull/5182","id":179632402,"number":5182,"title":"Metadata Allocation Classes","user":{"login":"don-brady","id":12415697,"avatar_url":"https://avatars1.githubusercontent.com/u/12415697?v=4","gravatar_id":"","url":"https://api.github.com/users/don-brady","html_url":"https://github.com/don-brady","followers_url":"https://api.github.com/users/don-brady/followers","following_url":"https://api.github.com/users/don-brady/following{/other_user}","gists_url":"https://api.github.com/users/don-brady/gists{/gist_id}","starred_url":"https://api.github.com/users/don-brady/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/don-brady/subscriptions","organizations_url":"https://api.github.com/users/don-brady/orgs","repos_url":"https://api.github.com/users/don-brady/repos","events_url":"https://api.github.com/users/don-brady/events{/privacy}","received_events_url":"https://api.github.com/users/don-brady/received_events","type":"User","site_admin":false},"labels":[{"id":40358,"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Feature","name":"Feature","color":"02d7e1","default":false},{"id":275713568,"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/OpenZFS%20Review","name":"OpenZFS Review","color":"eb6420","default":false},{"id":453022836,"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Work%20in%20Progress","name":"Work in Progress","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":66,"created_at":"2016-09-27T22:30:49Z","updated_at":"2017-12-09T06:57:50Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/zfsonlinux/zfs/pulls/5182","html_url":"https://github.com/zfsonlinux/zfs/pull/5182","diff_url":"https://github.com/zfsonlinux/zfs/pull/5182.diff","patch_url":"https://github.com/zfsonlinux/zfs/pull/5182.patch"},"body":"# Pool Allocation Classes [WIP]\r\n\r\n*Note:* there was a recent consolidation effort and the various allocation classes were merged into a single *special* class that serves metadata and small blocks allocations.\r\n\r\n## Allocation Class Categories\r\nAllocation classes can be thought of as allocation tiers that are dedicated to specific block categories.  These categories are differentiated by their function and listed below.\r\n### Block Class Categories\r\n- General data (classic default)\r\n- Intent Log data (SLOG)\r\n- Special (metadata, DDT, and blocks <= 32K)\r\n\r\n### Feature Flag Encapsulation\r\nThe `feature@allocation_classes` becomes active when a unique allocation class is instantiated by a VDEV.  Activating this feature makes the pool read-only on builds that don't support allocation classes.\r\n\r\n### Allocation Class Granularity\r\nThe class allocation bias occurs at a metaslab granularity. Each metaslab has an allocation bias which is assigned when the metaslab is initialized based on the VDEV's allocation bias.  The metaslab's allocation bias in turn determines which metaslab group to join.\r\n\r\n### Opt-in Variations\r\nThere are two _flavors_ for opting-in to allocation classes.\r\n* __Dedicated__ VDEVs\r\n* __Segregated__ VDEVs\r\nThese are described below and are mutually exclusive use cases (pick only one).\r\n\r\n## Dedicated VDEVs\r\nAll metaslabs in the VDEV are dedicated to a specific allocation class category. A pool must always have at least one general (non-specified) VDEV when using dedicated VDEVs. Opt-in using a VDEV class designation when creating the VDEV. Valid designation keywords are:\r\n\t `log | special`\r\n\r\n_Example Syntax:_\r\n\t`zpool create demo raidz <disks> special mirror <disks>`\r\n\t`zpool add demo special mirror <disks>`\r\n\r\n## Segregated VDEVs\r\nIn the segregated use case, a portion of a VDEV's metaslabs are set aside for a specific allocation class.\r\nOpt-in is global to the pool using a `boolean` pool property. These segregate properties can be combined if multiple segregation categories are desired (e.g.  segregate log class and segregate special class from normal class).\r\n```\r\nsegregate_log=on\r\nsegregate_special=on\r\n```\r\n_Example Syntax:_\r\n\t`zpool create -o segregate_special=on raidz <disks>`\r\n\r\n### Segregation Percentage\r\nBy default, the following segregation limits are applied:\r\n- `segregate_log` will set aside one metaslab per VDEV for log data.\r\n- `segregate_special` will set aside 20% of a VDEV's metaslabs for the special class\r\n\r\nThe segregated metaslabs are dynamically assigned using a first available algorithm when the VDEV is opened. Subsequent opens may shift which metaslabs are assigned, but once a metaslab is allocated from (i.e. it is activated), the preferred bias becomes persistent.\r\n\r\nBlock category allocation accounting can be observed from the CLI (see `zdb -mm` and `zpool list -C`).\r\n\r\n### Ditto Block Policy\r\nWhen there is only one VDEV available and more than one DVA is required (ditto copies > 1), the traditional ditto placement policy was to place the allocation a distance of 1/8th of total vdev allocation space away from the other DVAs.  This policy has been simplified to a simple guarantee that the other DVAs land in a different metaslab.  This in turn greatly simplifies ditto DVA placement from a segregated VDEV (where a group of metaslabs is not necessarily consecutive).  To validate that the new policy is honored, a zdb(8) block audit will also report any DVAs that landed in the same metaslab (expecting none). If there is a policy failure it will be manifest as follows in zdb output:\r\n`Dittoed blocks in same metaslab: 21`\r\n\r\n### VDEV Changes\r\nClasses show up in run-time VDEV instance as an allocation bias:\r\n```\r\n\ttypedef enum vdev_alloc_bias {\r\n\t        VDEV_BIAS_NONE,\r\n\t        VDEV_BIAS_LOG,          /* dedicated to ZIL data (SLOG) */\r\n\t        VDEV_BIAS_SPECIAL,    /* dedicated to metadata and small blocks */\r\n\t        VDEV_BIAS_SEGREGATE,    /* segregated metaslabs into multiple groups */\r\n\t} vdev_alloc_bias_t;\r\n```\r\nThis vdev allocation class bias information is stored in the per-vdev zap object as a string value:\r\n```\r\n\t/* vdev metaslab allocation bias */\r\n\t#define VDEV_ALLOC_BIAS_LOG             \"log\"\r\n\t#define VDEV_ALLOC_BIAS_SPECIAL        \"special\"\r\n\t#define VDEV_ALLOC_BIAS_SEGREGATE       \"segregate\"\r\n```\r\nThe bias is also passed internally in the pool config during a pool create and any internal pool config query.  This can be used by functions in the zpool(8) command.\r\n\r\nThere is additional VDEV metadata that is stored in the **VDEV_TOP_ZAP_METASLAB_INFO_OBJ** object. This object is an array of _ms_alloc_phys_ structures, one per metaslab, that tracks the allocation bias assigned to a metaslab and the allocation stats by category.\r\n```\r\n/*\r\n * Additional per-metaslab allocation info for dedicated/segregated vdevs\r\n */\r\ntypedef struct ms_alloc_phys {\r\n        uint64_t        ms_alloc_flags;         /* flags: ie segregated bias */\r\n        uint64_t        ms_alloc_metadata;      /* metadata space allocated */\r\n        uint64_t        ms_alloc_smallblks;     /* smallblks space allocated */\r\n        uint64_t        ms_alloc_dedup;         /* dedup space allocated */\r\n} ms_alloc_phys_t;\r\n\r\n/*\r\n * class allocation bias (segregated vdevs only)\r\n */\r\ntypedef enum {\r\n        MS_ALLOC_BIAS_UNASSIGNED =      0x00,\r\n        MS_ALLOC_BIAS_LOG =             0x01,\r\n        MS_ALLOC_BIAS_SPECIAL =         0x02,\r\n        MS_ALLOC_BIAS_NORMAL =          0x03\r\n} ms_alloc_bias_t;\r\n```\r\n\r\n#### VDEV Allocation Stats\r\nNew special class allocation info was added to `vdev_stat_t`.  This is primarily used by `zpool list -C` to query the allocation info by class category.  This information helps to determine if provisioning was correct.\r\n\r\n## Observing Allocation Classes\r\nThere are several ways to observe aspects of allocation classes using the zpool(8), zdb(8) and kstat\r\n### ZPOOL\r\n```\r\n\t$ zpool status demo\r\n\t  pool: demo\r\n\t state: ONLINE\r\n\t  scan: none requested\r\n\tconfig:\r\n\t        NAME                        STATE     READ WRITE CKSUM\r\n\t        demo                        ONLINE       0     0     0\r\n\t          raidz1-0                  ONLINE       0     0     0\r\n\t            scsi-35000c5007adc15a5  ONLINE       0     0     0\r\n\t            scsi-35000c5007adc6d2f  ONLINE       0     0     0\r\n\t            scsi-35000c5007adcf3f4  ONLINE       0     0     0\r\n\t            scsi-35000c5007add017e  ONLINE       0     0     0\r\n\t            scsi-35000c5007addaf56  ONLINE       0     0     0\r\n\t          special:mirror-1          ONLINE       0     0     0\r\n\t            wwn-0x55cd2e404c033dab  ONLINE       0     0     0\r\n\t            wwn-0x55cd2e404c033da3  ONLINE       0     0     0\r\n```\r\n```\r\n\t$ zpool list -o name,segregate_log,segregate_special\r\n\tNAME        SEGREGATE_LOG  SEGREGATE_SPECIAL\r\n\tdemo                  off                 on\r\n\tspill-test            off                off\r\n```\r\n```\t\r\n[root@ssu1_oss2]# zpool list -C ssu_1ost1\r\nNAME            SIZE   ALLOC  FREE   CAPACITY\r\n--------------  -----  -----  -----  --------\r\nssu_1ost1       72.7T  56.0T  16.8T  77.0%\r\n  draid2-0      72.7T  56.0T  16.8T  77.0%\r\n    normal      58.2T  55.8T  2.48T  95.8%\r\n    special     2.25T   217G  2.04T  9.43%\r\n    unassigned  12.2T      0  12.2T      -\r\n```\t\r\n\r\n### ZDB\r\nVarious portions of ZDB have been adapted to accommodate allocation class information\r\nlike `zdb -m`:\r\n* The dedicated case shows bias assigned to VDEV\r\n* The segregated case has an additional column for class (seen below)\r\n```\r\n\t$ sudo zdb -m demo\r\n\t\r\n\tMetaslabs:\r\n\t        vdev          0   segregate\r\n\t        metaslabs   145   offset                spacemap          free          class\r\n\t        ---------------   -------------------   ---------------   ------------  --------\r\n\t        metaslab      0   offset            0   spacemap     75   free   32.0G   special\r\n\t        metaslab      1   offset    800000000   spacemap     74   free   32.0G   special\r\n\t        metaslab      2   offset   1000000000   spacemap     73   free   32.0G   special\r\n\t        metaslab      3   offset   1800000000   spacemap      4   free   20.5G   special\r\n\t        metaslab      4   offset   2000000000   spacemap      3   free   31.7G   special\r\n\t        metaslab      5   offset   2800000000   spacemap      2   free   32.0G   special\r\n\t        metaslab      6   offset   3000000000   spacemap      9   free   20.3G   special\r\n\t        metaslab      7   offset   3800000000   spacemap      8   free   31.8G   special\r\n\t        metaslab      8   offset   4000000000   spacemap      7   free   32.0G   special\r\n\t        metaslab      9   offset   4800000000   spacemap     13   free   32.0G   special\r\n\t        metaslab     10   offset   5000000000   spacemap     12   free   32.0G   special\r\n\t        . . .\r\n\t        metaslab     29   offset   e800000000   spacemap      5   free   16.0G    normal\r\n\t        metaslab     30   offset   f000000000   spacemap      6   free   16.0G    normal\r\n\t        metaslab     31   offset   f800000000   spacemap     10   free   17.8G    normal\r\n\t        metaslab     32   offset  10000000000   spacemap      0   free     32G    normal\r\n\t        metaslab     33   offset  10800000000   spacemap      0   free     32G    normal\r\n\t        metaslab     34   offset  11000000000   spacemap      0   free     32G    normal\r\n\t        metaslab     35   offset  11800000000   spacemap      0   free     32G    normal\r\n\t        metaslab     36   offset  12000000000   spacemap      0   free     32G    normal\r\n\t        metaslab     37   offset  12800000000   spacemap      0   free     32G    normal\r\n\t        metaslab     38   offset  13000000000   spacemap      0   free     32G    normal\r\n\t        . . .\r\n\t        metaslab    144   offset  48000000000   spacemap      0   free     32G    normal\r\n```\t\r\n`zdb -mm [-P]` -- per metaslab allocation info by block categories:\r\n```\r\n\t        Allocation Summary:       33180160 allocated\r\n\t                   metadata:    2633216\r\n\t                  smallblks:   30546944\r\n\t                      dedup:          0\r\n\t                    generic:          0\r\n```\r\n`zdb -b <pool>` -- class summary and ditto policy (shows any same-vdev ditto blocks that landed in the same metaslab).\r\n```\r\n\t        No leaks (block sum matches space maps exactly)\r\n\t\r\n\t        bp count:                602424\r\n\t        ganged count:                 0\r\n\t        bp logical:         18845081600      avg:  31282\r\n\t        bp physical:        18475993600      avg:  30669     compression:   1.02\r\n\t        bp allocated:       19405235712      avg:  32211     compression:   0.97\r\n\t        bp deduped:                   0    ref>1:      0   deduplication:   1.00\r\n\t        Normal class:       18499772416     used:  4.63%\r\n\t        Special Class:      905463296     used: 87.05%\r\n\t\r\n\t        additional, non-pointer bps of type 0:        642\r\n\t        Dittoed blocks on same vdev: 27127\r\n```\r\n### KSTAT\r\n```\r\n[root@ssu1_oss2]# cat /proc/spl/kstat/zfs/alloc_class_stats\r\n \r\nname                            type data\r\nnormal_allocated                4    61325031915520\r\nnormal_highest_allocated        4    61325037486080\r\nspecial_allocated               4    233487310848\r\nspecial_highest_allocated       4    233536917504\r\nslog_allocated                  4    0\r\nslog_highest_allocated          4    0\r\n```\r\n\r\n## ZTEST Coverage\r\n* 20% of initial pools are created with vdev segregation enabled\r\n* Occasionally add a special dedicated vdev\r\n* Occasionally disable small blocks in special class\r\n* a new '-C' option allows control over dedicated and segregated vdev creation (see ztest(8) -?)\r\n\t\r\n## Caveats and TBD\r\n\r\n- Enabling the segregated VDEV feature is currently limited to pool create. The ability to enable it on an existing pool is a stated design goal but it requires testing that we haven't had time for.  It is prevented (by means of a set-once property) as a conservative measure in current builds.\r\n- The effect of causing parity hot spotting by the segregating of metadata away from file data is not known.\r\n- At this point there are no custom allocation policies and all classes use the default allocator.\r\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/350428887","html_url":"https://github.com/zfsonlinux/zfs/pull/5182#issuecomment-350428887","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5182","id":350428887,"user":{"login":"codecov[bot]","id":22429695,"avatar_url":"https://avatars2.githubusercontent.com/in/254?v=4","gravatar_id":"","url":"https://api.github.com/users/codecov%5Bbot%5D","html_url":"https://github.com/apps/codecov","followers_url":"https://api.github.com/users/codecov%5Bbot%5D/followers","following_url":"https://api.github.com/users/codecov%5Bbot%5D/following{/other_user}","gists_url":"https://api.github.com/users/codecov%5Bbot%5D/gists{/gist_id}","starred_url":"https://api.github.com/users/codecov%5Bbot%5D/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codecov%5Bbot%5D/subscriptions","organizations_url":"https://api.github.com/users/codecov%5Bbot%5D/orgs","repos_url":"https://api.github.com/users/codecov%5Bbot%5D/repos","events_url":"https://api.github.com/users/codecov%5Bbot%5D/events{/privacy}","received_events_url":"https://api.github.com/users/codecov%5Bbot%5D/received_events","type":"Bot","site_admin":false},"created_at":"2017-12-09T06:57:50Z","updated_at":"2017-12-09T06:57:50Z","author_association":"NONE","body":"# [Codecov](https://codecov.io/gh/zfsonlinux/zfs/pull/5182?src=pr&el=h1) Report\n> Merging [#5182](https://codecov.io/gh/zfsonlinux/zfs/pull/5182?src=pr&el=desc) into [master](https://codecov.io/gh/zfsonlinux/zfs/commit/94183a9d8a1133ff0d29666a86f84c24f2c4083c?src=pr&el=desc) will **increase** coverage by `0.15%`.\n> The diff coverage is `77.76%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/zfsonlinux/zfs/pull/5182/graphs/tree.svg?src=pr&width=650&token=NGfxvvG2io&height=150)](https://codecov.io/gh/zfsonlinux/zfs/pull/5182?src=pr&el=tree)\n\n```diff\n@@            Coverage Diff             @@\n##           master    #5182      +/-   ##\n==========================================\n+ Coverage    75.1%   75.26%   +0.15%     \n==========================================\n  Files         298      296       -2     \n  Lines       95270    96036     +766     \n==========================================\n+ Hits        71555    72279     +724     \n- Misses      23715    23757      +42\n```\n\n| Flag | Coverage Δ | |\n|---|---|---|\n| #kernel | `74.22% <64.15%> (-0.3%)` | :arrow_down: |\n| #user | `67.71% <74.44%> (+0.47%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/zfsonlinux/zfs/pull/5182?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/zfsonlinux/zfs/pull/5182?src=pr&el=footer). Last update [94183a9...6297e85](https://codecov.io/gh/zfsonlinux/zfs/pull/5182?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n"}}
{"actor":{"display_login":"codecov"}}