commit c00945eb8b1e8bc1a316771be374870167087cd8 Author: Vikhyat Umrao Date: Wed Jan 13 18:26:23 2016 +0530 cleanup: remove obsolete option "filestore_xattr_use_omap" This patch does cleanup for option "filestore_xattr_use_omap", as this option was removed in #7408. Fixes: #14356 Signed-off-by: Vikhyat Umrao diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst index b5a938f..7cf2555 100644 --- a/doc/rados/configuration/filesystem-recommendations.rst +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -45,12 +45,6 @@ does not bound the total xattr metadata stored with a file. ``XFS`` has a relatively large limit (64 KB) that most deployments won't encounter, but the ``ext4`` is too small to be usable. -You should always add the following line to the ``[osd]`` section of your -``ceph.conf`` file for ``ext4`` filesystems; you can optionally use -it for ``btrfs`` and ``XFS``.:: - - filestore xattr use omap = true - Filesystem Background Info ========================== commit 7f36312ab6757813fb04914a2cde60e37da89508 Author: Loic Dachary Date: Mon Mar 2 12:09:13 2015 +0100 doc: ext4 has a journal Signed-off-by: Loic Dachary diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst index fa60572..b5a938f 100644 --- a/doc/rados/configuration/filesystem-recommendations.rst +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -55,9 +55,10 @@ it for ``btrfs`` and ``XFS``.:: Filesystem Background Info ========================== -The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly -scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``. -Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that +The ``XFS``, ``btrfs`` and ``ext4`` file systems provide numerous advantages in highly +scaled data storage environments when `compared`_ to ``ext3``. + +``XFS``, ``btrfs`` and ``ext4`` are `journaling file systems`_, which means that they are more robust when recovering from crashes, power outages, etc. These filesystems journal all of the changes they will make before performing writes. commit 3f3ad61fae2af362310ca01884f6f15d48547feb Author: John Wilkins Date: Tue Jun 11 12:10:52 2013 -0700 doc: Fixed :term" syntax. Signed-off-by: John Wilkins diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst index 27d2aa9..fa60572 100644 --- a/doc/rados/configuration/filesystem-recommendations.rst +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -16,7 +16,7 @@ Use ``hdparm`` to disable write caching on the hard disk:: sudo hdparm -W 0 /dev/hda 0 -In production environments, we recommend running :term:`Ceph OSD Daemons` with +In production environments, we recommend running a :term:`Ceph OSD Daemon` with separate drives for the operating system and the data. If you run data and an operating system on a single disk, we recommend creating a separate partition for your data. commit dea8c2d1889ec302f2aa0c3ba2ac417dcacb28aa Author: John Wilkins Date: Fri Jun 7 09:51:05 2013 -0700 doc: Updated for glossary terms and added indexing. Signed-off-by: John Wilkins diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst index cb1aca8..27d2aa9 100644 --- a/doc/rados/configuration/filesystem-recommendations.rst +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -2,29 +2,32 @@ Hard Disk and File System Recommendations =========================================== -Hard Disk Prep -============== +.. index:: hard drive preparation -Ceph aims for data safety, which means that when the application receives notice -that data was written to the disk, that data was actually written to the disk. -For old kernels (<2.6.33), disable the write cache if the journal is on a raw -disk. Newer kernels should work fine. +Hard Drive Prep +=============== + +Ceph aims for data safety, which means that when the :term:`Ceph Client` +receives notice that data was written to a storage drive, that data was actually +written to the storage drive. For old kernels (<2.6.33), disable the write cache +if the journal is on a raw drive. Newer kernels should work fine. Use ``hdparm`` to disable write caching on the hard disk:: sudo hdparm -W 0 /dev/hda 0 -In production environments, we recommend running OSDs with an operating system -disk, and a separate disk(s) for data. If you run data and an operating system -on a single disk, create a separate partition for your data before configuring -your OSD cluster. +In production environments, we recommend running :term:`Ceph OSD Daemons` with +separate drives for the operating system and the data. If you run data and an +operating system on a single disk, we recommend creating a separate partition +for your data. +.. index:: filesystems -File Systems -============ +Filesystems +=========== -Ceph OSDs rely heavily upon the stability and performance of the -underlying file system. +Ceph OSD Daemons rely heavily upon the stability and performance of the +underlying filesystem. .. note:: We currently recommend ``XFS`` for production deployments. We recommend ``btrfs`` for testing, development, and any @@ -35,13 +38,12 @@ underlying file system. comfortable installing the latest released upstream kernels and be able to track development activity for critical bug fixes. -Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying -file system for various forms of internal object state and metadata. -The underlying file system must provide sufficient capacity for -XATTRs. ``btrfs`` does not bound the total xattr metadata stored with -a file. ``XFS`` has a relatively large limit (64 KB) that most -deployments won't encounter, but the ``ext4`` is too small to be -usable. +Ceph OSD Daemons depend on the Extended Attributes (XATTRs) of the underlying +file system for various forms of internal object state and metadata. The +underlying filesystem must provide sufficient capacity for XATTRs. ``btrfs`` +does not bound the total xattr metadata stored with a file. ``XFS`` has a +relatively large limit (64 KB) that most deployments won't encounter, but the +``ext4`` is too small to be usable. You should always add the following line to the ``[osd]`` section of your ``ceph.conf`` file for ``ext4`` filesystems; you can optionally use @@ -49,8 +51,9 @@ it for ``btrfs`` and ``XFS``.:: filestore xattr use omap = true -FS Background Info -================== + +Filesystem Background Info +========================== The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``. commit 37b57cdf0fdc5c03eeff3f5eb58ff4010ce581f6 Author: rca Date: Thu Jan 3 13:30:01 2013 -0800 Update doc/rados/configuration/filesystem-recommendations.rst Clarified when it's necessary to use the setting: filestore xattr use omap = true diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst index 17908cc..cb1aca8 100644 --- a/doc/rados/configuration/filesystem-recommendations.rst +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -41,8 +41,11 @@ The underlying file system must provide sufficient capacity for XATTRs. ``btrfs`` does not bound the total xattr metadata stored with a file. ``XFS`` has a relatively large limit (64 KB) that most deployments won't encounter, but the ``ext4`` is too small to be -usable. To use these file systems, you should add the following like -to the ``[osd]`` section of your ``ceph.conf`` file.:: +usable. + +You should always add the following line to the ``[osd]`` section of your +``ceph.conf`` file for ``ext4`` filesystems; you can optionally use +it for ``btrfs`` and ``XFS``.:: filestore xattr use omap = true commit 0a2a0c075c32f541c8746530690788adaf048b78 Author: John Wilkins Date: Wed Nov 14 14:57:51 2012 -0800 doc: config-cluser move to new IA. Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst similarity index 100% rename from doc/config-cluster/file-system-recommendations.rst rename to doc/rados/configuration/filesystem-recommendations.rst commit a1b2f584eb08031d411247b943f1a8eefb4dee2f Author: John Wilkins Date: Wed Nov 7 14:07:28 2012 -0800 doc: Fix and minor edit for admonitions. Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index 06971c8..17908cc 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -26,15 +26,14 @@ File Systems Ceph OSDs rely heavily upon the stability and performance of the underlying file system. -.. tip:: We currently recommend ``XFS`` for production deployments. - -.. tip:: We recommend ``btrfs`` for testing, development, and any -non-critical deployments. We believe that ``btrfs`` has the correct -feature set and roadmap to serve Ceph in the long-term, but ``XFS`` -and ``ext4`` provide the necessary stability for today's deployments. -``btrfs`` development is proceeding rapidly: users should be -comfortable installing the latest released upstream kernels and be -able to track development activity for critical bug fixes. +.. note:: We currently recommend ``XFS`` for production deployments. + We recommend ``btrfs`` for testing, development, and any + non-critical deployments. We believe that ``btrfs`` has the correct + feature set and roadmap to serve Ceph in the long-term, but ``XFS`` + and ``ext4`` provide the necessary stability for today's deployments. + ``btrfs`` development is proceeding rapidly: users should be + comfortable installing the latest released upstream kernels and be + able to track development activity for critical bug fixes. Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for various forms of internal object state and metadata. commit 3a48cbf245f36306052d004b2a7411ea9d803cd0 Author: Sage Weil Date: Mon Oct 29 13:01:06 2012 -0700 doc: update fs recommendations More forceful about recommending XFS. More warning about using btrfs in production deployments. Signed-off-by: Sage Weil diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index e56943e..06971c8 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -23,37 +23,30 @@ your OSD cluster. File Systems ============ -Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file -system for: - -- Internal object state -- Snapshot metadata -- RADOS Gateway Access Control Lists (ACLs). - -Ceph OSDs rely heavily upon the stability and performance of the underlying file -system. The underlying file system must provide sufficient capacity for XATTRs. -File system candidates for Ceph include B tree and B+ tree file systems such as: - -- ``btrfs`` -- ``XFS`` - -If you are using ``ext4``, mount your file system to enable XATTRs. You must also -add the following line to the ``[osd]`` section of your ``ceph.conf`` file. :: +Ceph OSDs rely heavily upon the stability and performance of the +underlying file system. + +.. tip:: We currently recommend ``XFS`` for production deployments. + +.. tip:: We recommend ``btrfs`` for testing, development, and any +non-critical deployments. We believe that ``btrfs`` has the correct +feature set and roadmap to serve Ceph in the long-term, but ``XFS`` +and ``ext4`` provide the necessary stability for today's deployments. +``btrfs`` development is proceeding rapidly: users should be +comfortable installing the latest released upstream kernels and be +able to track development activity for critical bug fixes. + +Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying +file system for various forms of internal object state and metadata. +The underlying file system must provide sufficient capacity for +XATTRs. ``btrfs`` does not bound the total xattr metadata stored with +a file. ``XFS`` has a relatively large limit (64 KB) that most +deployments won't encounter, but the ``ext4`` is too small to be +usable. To use these file systems, you should add the following like +to the ``[osd]`` section of your ``ceph.conf`` file.:: filestore xattr use omap = true -.. tip:: Use ``xfs`` initially and ``btrfs`` when it is ready for production. - - The Ceph team believes that the best performance and stability will come from - ``btrfs.`` The ``btrfs`` file system has internal transactions that keep the - local data set in a consistent state. This makes OSDs based on ``btrfs`` simple - to deploy, while providing scalability not currently available from block-based - file systems. The 64-kb XATTR limit for ``xfs`` XATTRS is enough to accommodate - RDB snapshot metadata and RADOS Gateway ACLs. So ``xfs`` is the second-choice - file system of the Ceph team in the long run, but ``xfs`` is currently more - stable than ``btrfs``. If you only plan to use RADOS and ``rbd`` without - snapshots and without ``radosgw``, the ``ext4`` file system should work just fine. - FS Background Info ================== commit 168bd10e4df24baaa2290aff7a6b6b5a1eba1841 Author: Sage Weil Date: Sat Oct 13 20:12:48 2012 -0700 doc: fix file system recs - drop xattr warning; this is not an issue with the leveldb stuff. - the ext3 vs xattr discussion was somewhat inaccurate. also, no longer relevant. Signed-off-by: Sage Weil diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index b498dc5..e56943e 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -42,14 +42,6 @@ add the following line to the ``[osd]`` section of your ``ceph.conf`` file. :: filestore xattr use omap = true -.. warning:: XATTR limits. - - The RADOS Gateway's ACL and Ceph snapshots easily surpass the 4-kilobyte limit - for XATTRs in ``ext4``, causing the ``ceph-osd`` process to crash. Version 0.45 - or newer uses ``leveldb`` to bypass this limitation. ``ext4`` is a poor file - system choice if you intend to deploy the RADOS Gateway or use snapshots on - versions earlier than 0.45. - .. tip:: Use ``xfs`` initially and ``btrfs`` when it is ready for production. The Ceph team believes that the best performance and stability will come from @@ -65,13 +57,6 @@ add the following line to the ``[osd]`` section of your ``ceph.conf`` file. :: FS Background Info ================== -Before ``ext3``, ``ReiserFS`` was the only journaling file system available for -Linux. However, ``ext3`` doesn't provide Extended Attribute (XATTR) support. -While ``ext4`` provides XATTR support, it only allows XATTRs up to 4kb. The -4kb limit is not enough for RADOS GW ACLs, snapshots, and other features. As of -version 0.45, Ceph provides a ``leveldb`` feature for ``ext4`` file systems -that stores XATTRs in excess of 4kb in a ``leveldb`` database. - The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``. Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that commit d90fea6cadbbb74cef60642ae153925aa07000a8 Author: John Wilkins Date: Wed Sep 5 17:21:04 2012 -0700 :doc: Consolidated file system recommendations. Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index 93f52d6..b498dc5 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -2,6 +2,9 @@ Hard Disk and File System Recommendations =========================================== +Hard Disk Prep +============== + Ceph aims for data safety, which means that when the application receives notice that data was written to the disk, that data was actually written to the disk. For old kernels (<2.6.33), disable the write cache if the journal is on a raw @@ -16,6 +19,10 @@ disk, and a separate disk(s) for data. If you run data and an operating system on a single disk, create a separate partition for your data before configuring your OSD cluster. + +File Systems +============ + Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for: @@ -54,3 +61,41 @@ add the following line to the ``[osd]`` section of your ``ceph.conf`` file. :: file system of the Ceph team in the long run, but ``xfs`` is currently more stable than ``btrfs``. If you only plan to use RADOS and ``rbd`` without snapshots and without ``radosgw``, the ``ext4`` file system should work just fine. + +FS Background Info +================== + +Before ``ext3``, ``ReiserFS`` was the only journaling file system available for +Linux. However, ``ext3`` doesn't provide Extended Attribute (XATTR) support. +While ``ext4`` provides XATTR support, it only allows XATTRs up to 4kb. The +4kb limit is not enough for RADOS GW ACLs, snapshots, and other features. As of +version 0.45, Ceph provides a ``leveldb`` feature for ``ext4`` file systems +that stores XATTRs in excess of 4kb in a ``leveldb`` database. + +The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly +scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``. +Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that +they are more robust when recovering from crashes, power outages, etc. These +filesystems journal all of the changes they will make before performing writes. + +``XFS`` was developed for Silicon Graphics, and is a mature and stable +filesystem. By contrast, ``btrfs`` is a relatively new file system that aims +to address the long-standing wishes of system administrators working with +large scale data storage environments. ``btrfs`` has some unique features +and advantages compared to other Linux filesystems. + +``btrfs`` is a `copy-on-write`_ filesystem. It supports file creation +timestamps and checksums that verify metadata integrity, so it can detect +bad copies of data and fix them with the good copies. The copy-on-write +capability means that ``btrfs`` can support snapshots that are writable. +``btrfs`` supports transparent compression and other features. + +``btrfs`` also incorporates multi-device management into the file system, +which enables you to support heterogeneous disk storage infrastructure, +data allocation policies. The community also aims to provide ``fsck``, +deduplication, and data encryption support in the future. This compelling +list of features makes ``btrfs`` the ideal choice for Ceph clusters. + +.. _copy-on-write: http://en.wikipedia.org/wiki/Copy-on-write +.. _compared: http://en.wikipedia.org/wiki/Comparison_of_file_systems +.. _journaling file systems: http://en.wikipedia.org/wiki/Journaling_file_system commit a466dfff2d2002f2b85ebc1b81167ee7d5c53f30 Author: John Wilkins Date: Tue Aug 28 11:02:13 2012 -0700 doc: added sudo for hdparm command. Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index 82972d7..93f52d6 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -9,7 +9,7 @@ disk. Newer kernels should work fine. Use ``hdparm`` to disable write caching on the hard disk:: - hdparm -W 0 /dev/hda 0 + sudo hdparm -W 0 /dev/hda 0 In production environments, we recommend running OSDs with an operating system disk, and a separate disk(s) for data. If you run data and an operating system commit 8869621fdf1c3bea44d374170e0e06cc96a7b4c4 Author: John Wilkins Date: Tue Jul 10 08:14:42 2012 -0700 doc: Added some pre-clarification for gdisk. Added DHO OSD hardware config. Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index 5049f38..82972d7 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -11,6 +11,10 @@ Use ``hdparm`` to disable write caching on the hard disk:: hdparm -W 0 /dev/hda 0 +In production environments, we recommend running OSDs with an operating system +disk, and a separate disk(s) for data. If you run data and an operating system +on a single disk, create a separate partition for your data before configuring +your OSD cluster. Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for: commit 812989bf35d18416b494c06943ecc74a1bddcc27 Author: John Wilkins Date: Fri May 18 13:54:51 2012 -0700 doc: misc updates doc/architecture.rst - removed broken reference. doc/config-cluster - cleanup and added chef doc/install - Made generic to add Chef, OpenStack and libvert installs doc/init - Created light start | stop and health section doc/source - Removed $ from code examples. Trimmed paras to 80 char doc/images - Added preliminary diagram for Chef. doc/rec - Added reference to hardware. Added filesystem info. Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index 6ea4950..5049f38 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -1,6 +1,6 @@ -========================================= -Hard Disk and File System Recommendations -========================================= +=========================================== + Hard Disk and File System Recommendations +=========================================== Ceph aims for data safety, which means that when the application receives notice that data was written to the disk, that data was actually written to the disk. @@ -9,7 +9,7 @@ disk. Newer kernels should work fine. Use ``hdparm`` to disable write caching on the hard disk:: - $ hdparm -W 0 /dev/hda 0 + hdparm -W 0 /dev/hda 0 Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file @@ -26,7 +26,8 @@ File system candidates for Ceph include B tree and B+ tree file systems such as: - ``btrfs`` - ``XFS`` -If you are using ``ext4``, enable XATTRs. :: +If you are using ``ext4``, mount your file system to enable XATTRs. You must also +add the following line to the ``[osd]`` section of your ``ceph.conf`` file. :: filestore xattr use omap = true commit 5465e810977d5621f107bf1fea1a4740e11a3290 Author: Tommi Virtanen Date: Thu May 3 10:15:21 2012 -0700 doc: Whitespace cleanup. Signed-off-by: Tommi Virtanen diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst index 4af540d..6ea4950 100644 --- a/doc/config-cluster/file-system-recommendations.rst +++ b/doc/config-cluster/file-system-recommendations.rst @@ -2,26 +2,26 @@ Hard Disk and File System Recommendations ========================================= -Ceph aims for data safety, which means that when the application receives notice -that data was written to the disk, that data was actually written to the disk. -For old kernels (<2.6.33), disable the write cache if the journal is on a raw +Ceph aims for data safety, which means that when the application receives notice +that data was written to the disk, that data was actually written to the disk. +For old kernels (<2.6.33), disable the write cache if the journal is on a raw disk. Newer kernels should work fine. -Use ``hdparm`` to disable write caching on the hard disk:: +Use ``hdparm`` to disable write caching on the hard disk:: - $ hdparm -W 0 /dev/hda 0 + $ hdparm -W 0 /dev/hda 0 - -Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file + +Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file system for: - Internal object state - Snapshot metadata -- RADOS Gateway Access Control Lists (ACLs). +- RADOS Gateway Access Control Lists (ACLs). Ceph OSDs rely heavily upon the stability and performance of the underlying file -system. The underlying file system must provide sufficient capacity for XATTRs. -File system candidates for Ceph include B tree and B+ tree file systems such as: +system. The underlying file system must provide sufficient capacity for XATTRs. +File system candidates for Ceph include B tree and B+ tree file systems such as: - ``btrfs`` - ``XFS`` @@ -34,19 +34,18 @@ If you are using ``ext4``, enable XATTRs. :: The RADOS Gateway's ACL and Ceph snapshots easily surpass the 4-kilobyte limit for XATTRs in ``ext4``, causing the ``ceph-osd`` process to crash. Version 0.45 - or newer uses ``leveldb`` to bypass this limitation. ``ext4`` is a poor file - system choice if you intend to deploy the RADOS Gateway or use snapshots on - versions earlier than 0.45. - + or newer uses ``leveldb`` to bypass this limitation. ``ext4`` is a poor file + system choice if you intend to deploy the RADOS Gateway or use snapshots on + versions earlier than 0.45. + .. tip:: Use ``xfs`` initially and ``btrfs`` when it is ready for production. - The Ceph team believes that the best performance and stability will come from - ``btrfs.`` The ``btrfs`` file system has internal transactions that keep the - local data set in a consistent state. This makes OSDs based on ``btrfs`` simple - to deploy, while providing scalability not currently available from block-based - file systems. The 64-kb XATTR limit for ``xfs`` XATTRS is enough to accommodate - RDB snapshot metadata and RADOS Gateway ACLs. So ``xfs`` is the second-choice - file system of the Ceph team in the long run, but ``xfs`` is currently more - stable than ``btrfs``. If you only plan to use RADOS and ``rbd`` without + The Ceph team believes that the best performance and stability will come from + ``btrfs.`` The ``btrfs`` file system has internal transactions that keep the + local data set in a consistent state. This makes OSDs based on ``btrfs`` simple + to deploy, while providing scalability not currently available from block-based + file systems. The 64-kb XATTR limit for ``xfs`` XATTRS is enough to accommodate + RDB snapshot metadata and RADOS Gateway ACLs. So ``xfs`` is the second-choice + file system of the Ceph team in the long run, but ``xfs`` is currently more + stable than ``btrfs``. If you only plan to use RADOS and ``rbd`` without snapshots and without ``radosgw``, the ``ext4`` file system should work just fine. - commit 93dcc9886fb833e987efd7eaf3bb71d84d79eda9 Author: Tommi Virtanen Date: Thu May 3 10:10:29 2012 -0700 doc: Rename to use dashes not underscores in URLs. This makes the-separate-words in the url match as separate words in searches, where this_way only matches an explicit "this_way" search. http://www.mattcutts.com/blog/dashes-vs-underscores/ Signed-off-by: Tommi Virtanen diff --git a/doc/config-cluster/file_system_recommendations.rst b/doc/config-cluster/file-system-recommendations.rst similarity index 100% rename from doc/config-cluster/file_system_recommendations.rst rename to doc/config-cluster/file-system-recommendations.rst commit 715a69c1ed5a16b0cbcc186ac956c6d5a13fef37 Author: Tommi Virtanen Date: Thu May 3 10:02:29 2012 -0700 doc: Remove leading whitespace, that syntax means it's a definition list. Signed-off-by: Tommi Virtanen diff --git a/doc/config-cluster/file_system_recommendations.rst b/doc/config-cluster/file_system_recommendations.rst index f2411b6..4af540d 100644 --- a/doc/config-cluster/file_system_recommendations.rst +++ b/doc/config-cluster/file_system_recommendations.rst @@ -20,7 +20,7 @@ system for: - RADOS Gateway Access Control Lists (ACLs). Ceph OSDs rely heavily upon the stability and performance of the underlying file - system. The underlying file system must provide sufficient capacity for XATTRs. +system. The underlying file system must provide sufficient capacity for XATTRs. File system candidates for Ceph include B tree and B+ tree file systems such as: - ``btrfs`` commit d49c3d29a7bb5c36c9d6e2b67b2306e93f5eddf4 Author: John Wilkins Date: Wed May 2 20:31:35 2012 -0700 Removed "Ceph Development Status" per Bryan Modified title syntax per Tommi Modified paragraph width to 80-chars per Dan Moved "Build from Source" out of Install Renamed create_cluster to config-cluster Added config-ref with configuration reference tables Added a toc ref for man/1/obsync per Dan Removed redundant sections from Ops Deleted "Why use Ceph" and "Introduction to Storage Clusters" Signed-off-by: John Wilkins diff --git a/doc/config-cluster/file_system_recommendations.rst b/doc/config-cluster/file_system_recommendations.rst new file mode 100644 index 0000000..f2411b6 --- /dev/null +++ b/doc/config-cluster/file_system_recommendations.rst @@ -0,0 +1,52 @@ +========================================= +Hard Disk and File System Recommendations +========================================= + +Ceph aims for data safety, which means that when the application receives notice +that data was written to the disk, that data was actually written to the disk. +For old kernels (<2.6.33), disable the write cache if the journal is on a raw +disk. Newer kernels should work fine. + +Use ``hdparm`` to disable write caching on the hard disk:: + + $ hdparm -W 0 /dev/hda 0 + + +Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file +system for: + +- Internal object state +- Snapshot metadata +- RADOS Gateway Access Control Lists (ACLs). + +Ceph OSDs rely heavily upon the stability and performance of the underlying file + system. The underlying file system must provide sufficient capacity for XATTRs. +File system candidates for Ceph include B tree and B+ tree file systems such as: + +- ``btrfs`` +- ``XFS`` + +If you are using ``ext4``, enable XATTRs. :: + + filestore xattr use omap = true + +.. warning:: XATTR limits. + + The RADOS Gateway's ACL and Ceph snapshots easily surpass the 4-kilobyte limit + for XATTRs in ``ext4``, causing the ``ceph-osd`` process to crash. Version 0.45 + or newer uses ``leveldb`` to bypass this limitation. ``ext4`` is a poor file + system choice if you intend to deploy the RADOS Gateway or use snapshots on + versions earlier than 0.45. + +.. tip:: Use ``xfs`` initially and ``btrfs`` when it is ready for production. + + The Ceph team believes that the best performance and stability will come from + ``btrfs.`` The ``btrfs`` file system has internal transactions that keep the + local data set in a consistent state. This makes OSDs based on ``btrfs`` simple + to deploy, while providing scalability not currently available from block-based + file systems. The 64-kb XATTR limit for ``xfs`` XATTRS is enough to accommodate + RDB snapshot metadata and RADOS Gateway ACLs. So ``xfs`` is the second-choice + file system of the Ceph team in the long run, but ``xfs`` is currently more + stable than ``btrfs``. If you only plan to use RADOS and ``rbd`` without + snapshots and without ``radosgw``, the ``ext4`` file system should work just fine. +