Where can directFS be downloaded? This is open source, right? On Tue, Jun 10, 2014 at 5:18 PM, Daniel Black wrote:

Thanks Colin. Missed that line of the doco and/or assumed it was a capability of another file-system.

----- Original Message -----

...

Special ioctl doesn't work unless you use directfs (you're using ext4); see: https://mariadb.com/kb/en/fusionio-directfs-atomic-write-support/

On Wednesday, 11 June 2014, Daniel Black < daniel.black@openquery.com

...

wrote:

Hi,

I'm trying to get innodb_use_atomic_writes = 1 working on an ext4 on top of LVM on a ioDrive device with a RHEL6 kernel 2.6.32-358.6.2.el6.x86_64

and getting the following error. Which bit(s) need to change to get it working?

140611 7:43:46 InnoDB: Operating system error number 25 in a file operation. InnoDB: Error number 25 means 'Inappropriate ioctl for device'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html InnoDB: File name ./ibdata1 InnoDB: File operation call: 'ioctl'. InnoDB: Error in opening ./ibdata1

-- -- Daniel Black, Engineer @ Open Query ( http://openquery.com.au ) Remote expertise & maintenance for MySQL/MariaDB server environments.

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- Mark Callaghan mdcallag@gmail.com

----- Original Message -----

Hi Mark,

On 14 Jun 2014, at 00:44, MARK CALLAGHAN wrote:

...

Where can directFS be downloaded? This is open source, right?

From what I gather, there is a public API to this called OpenNVM (http://opennvm.github.io/). But the short answer is that directfs/nvmfs isn't opensource - you only get this if you have a fusion-io card

Further more after talking to a fusionio engineer directfs development has stopped 2 years ago and is unsupported by them. opennvm was mentioned as its replacement though I've got him digging up some closed source stuff too. Lets see if I get a nvmfs. Despite the opennvm.github.io claims it looks like mariadb doesn't support it as it isn't a filesystem nor does it implement some ioctl, its a pure API that directly operates on the /dev/fio* device. Seems like some documentation is quite out of date. I'll let you know how I get on in about a week.

Fusion-io itself is a commercial product, so nvmfs/directfs is likely from them only (and is probably closed source). I take it others can implement it, but haven't (yet?) ?

MariaDB's initial implementation of it is here: http://lists.askmonty.org/pipermail/commits/2013-March/004529.html (and I presume this was then ported to Percona Server as well -- http://www.percona.com/doc/percona-server/5.6/performance/atomic_fio.html)

It seems the magic is at: #define DFS_IOCTL_ATOMIC_WRITE_SET _IOW(0x95, 2, uint)

I found a presentation that talks about it (nothing about the filesystem itself being opensource): http://www.slideshare.net/fusionio/perconalive-accelerating-mysql-in-open-so...

And for reference some older blog posts: https://blog.mariadb.org/significant-performance-boost-with-new-mariadb-page... https://blog.mariadb.org/mariadb-introduces-atomic-writes/

And attached for brevity, is a short chat I had with Jan Lindström who works on the fusion-io code now:

11:42 < bytee> jplindst: our fusion-io work, is based on the open APIs at http://opennvm.github.io/ correct? 11:46 < jplindst> bytee: To be totally honest, I have no idea what are the API at the http page you mentioned, I have used standard POSIX interfaces to access FusionIO storage, thus you will need FusionIO SSD card and NVMFS file system 11:47 < bytee> jplindst: nvmfs file system - where does one get it ? from fusion-io ? 11:47 < jplindst> bytee: FusionIO card is naturally commercial product, nvmfs to current my understanding does not work with any other device 11:48 < bytee> jplindst: ok, likely because others haven't implemented it 11:48 < jplindst> bytee: correct 11:49 < jplindst> bytee: If you have the card you should have the file system module 11:50 < bytee> jplindst: thanks 11:51 < jplindst> bytee: Last time we did discuss with FusionIO the actual file system was not yet open source 11:51 < jplindst> bytee: not sure if that has been changed now that announcements are out 11:51 < bytee> jplindst: doubt there is, from what i gathere

I hope this helps

cheers, -colin

...

On Tue, Jun 10, 2014 at 5:18 PM, Daniel Black wrote:

Thanks Colin. Missed that line of the doco and/or assumed it was a capability of another file-system.

----- Original Message -----

...

Special ioctl doesn't work unless you use directfs (you're using ext4); see: https://mariadb.com/kb/en/fusionio-directfs-atomic-write-support/

On Wednesday, 11 June 2014, Daniel Black < daniel.black@openquery.com

...

wrote:

Hi,

I'm trying to get innodb_use_atomic_writes = 1 working on an ext4 on top of LVM on a ioDrive device with a RHEL6 kernel 2.6.32-358.6.2.el6.x86_64

and getting the following error. Which bit(s) need to change to get it working?

140611 7:43:46 InnoDB: Operating system error number 25 in a file operation. InnoDB: Error number 25 means 'Inappropriate ioctl for device'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html InnoDB: File name ./ibdata1 InnoDB: File operation call: 'ioctl'. InnoDB: Error in opening ./ibdata1

-- -- Daniel Black, Engineer @ Open Query ( http://openquery.com.au ) Remote expertise & maintenance for MySQL/MariaDB server environments.

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- Mark Callaghan mdcallag@gmail.com

-- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

-- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

Just to add some flavor. NVM (also known as page) compression does indeed work with any file system supporting the fallocate/trim/discard commands passed via user space (EXT4, XFS). Atomics (as far as I know) is currently only implemented by NVMFS but is not a requirement for NVM compression. It will speed it up however. The first atomics T10 committee passed the first atomic SCSI command very recently: http://www.t10.org/cgi-bin/ac.pl?t=d&f=14-043r4.pdf Atomics will also help workloads not using compression. Mainly when there’s a certain amount of memory pressure inside InnoDB and heavy LRU flushing. To make it simple initially, InnoDB just issues an ioctl() to enable atomics on a per file basis. I would expect O_ATOMIC to be the way of doing it going forward once this matures. The first code for that support is here: http://lwn.net/Articles/573092/ All of the presentations and published numbers of NVM compression we have published has been with NVMFS+atomics and that package is available for testing right now on MariaDB10 among others. Hope this helps, Torben On Jun 18, 2014, at 12:58 AM, Dhananjoy Das wrote:

+ Nich Piggin, Torben

Hi Daniel, This is Dhananjoy Das, I can be blamed for the NVM Compression, co-authored with MariaDB :).

To help answer for the status of NVMFS, I can confirm NVMFS is a fully POSIX compliant filesystem and currently is supported for EA (early access) program. NVMFS is actively being developed for addressing interesting workloads. Feel free to send out any queries you may have on NVMFS to Nick/Torben (cc-ed). Nick, BTW is the author of the NVMFS filesystem.

Opennvm is an independent project/effort independent of NVMFS, sorry if any of the blogs has created any confusion. If you need details specifically on opennvm send it my way and I should be able to find an appropriate contact in fusion-io to help answer.

NVM-Compression uses only POSIX interfaces for its functionality (fallocate for preallocate/TRIMs), and the IOCTL (DFS_IOCTL_ATOMIC_WRITE_SET _IOW) is only used to enable transparent ATOMIC-writes (invoked via io_submit()) on the specified file.

Hope the information help, please feel free to reach out directly.

Thanks, Dhananjoy Das (Joy)

On 6/17/14, 3:04 PM, "Daniel Black" wrote:

...

----- Original Message -----

...

Hi Mark,

On 14 Jun 2014, at 00:44, MARK CALLAGHAN wrote:

...

Where can directFS be downloaded? This is open source, right?

From what I gather, there is a public API to this called OpenNVM (http://opennvm.github.io/). But the short answer is that directfs/nvmfs isn't opensource - you only get this if you have a fusion-io card

Further more after talking to a fusionio engineer directfs development has stopped 2 years ago and is unsupported by them.

opennvm was mentioned as its replacement though I've got him digging up some closed source stuff too. Lets see if I get a nvmfs.

Despite the opennvm.github.io claims it looks like mariadb doesn't support it as it isn't a filesystem nor does it implement some ioctl, its a pure API that directly operates on the /dev/fio* device.

Seems like some documentation is quite out of date.

I'll let you know how I get on in about a week.

...

Fusion-io itself is a commercial product, so nvmfs/directfs is likely from them only (and is probably closed source). I take it others can implement it, but haven't (yet?) ?

MariaDB's initial implementation of it is here: http://lists.askmonty.org/pipermail/commits/2013-March/004529.html (and I presume this was then ported to Percona Server as well --

http://www.percona.com/doc/percona-server/5.6/performance/atomic_fio.html )

It seems the magic is at: #define DFS_IOCTL_ATOMIC_WRITE_SET _IOW(0x95, 2, uint)

I found a presentation that talks about it (nothing about the filesystem itself being opensource):

http://www.slideshare.net/fusionio/perconalive-accelerating-mysql-in-open -source-hyperscale-systems

And for reference some older blog posts:

https://blog.mariadb.org/significant-performance-boost-with-new-mariadb-p age-compression-on-fusionio/ https://blog.mariadb.org/mariadb-introduces-atomic-writes/

And attached for brevity, is a short chat I had with Jan Lindström who works on the fusion-io code now:

11:42 < bytee> jplindst: our fusion-io work, is based on the open APIs at http://opennvm.github.io/ correct? 11:46 < jplindst> bytee: To be totally honest, I have no idea what are the API at the http page you mentioned, I have used standard POSIX interfaces to access FusionIO storage, thus you will need FusionIO SSD card and NVMFS file system 11:47 < bytee> jplindst: nvmfs file system - where does one get it ? from fusion-io ? 11:47 < jplindst> bytee: FusionIO card is naturally commercial product, nvmfs to current my understanding does not work with any other device 11:48 < bytee> jplindst: ok, likely because others haven't implemented it 11:48 < jplindst> bytee: correct 11:49 < jplindst> bytee: If you have the card you should have the file system module 11:50 < bytee> jplindst: thanks 11:51 < jplindst> bytee: Last time we did discuss with FusionIO the actual file system was not yet open source 11:51 < jplindst> bytee: not sure if that has been changed now that announcements are out 11:51 < bytee> jplindst: doubt there is, from what i gathere

I hope this helps

cheers, -colin

...

On Tue, Jun 10, 2014 at 5:18 PM, Daniel Black wrote:

Thanks Colin. Missed that line of the doco and/or assumed it was a capability of another file-system.

----- Original Message -----

...

Special ioctl doesn't work unless you use directfs (you're using ext4); see: https://mariadb.com/kb/en/fusionio-directfs-atomic-write-support/

On Wednesday, 11 June 2014, Daniel Black < daniel.black@openquery.com

...

wrote:

Hi,

I'm trying to get innodb_use_atomic_writes = 1 working on an ext4 on top of LVM on a ioDrive device with a RHEL6 kernel 2.6.32-358.6.2.el6.x86_64

and getting the following error. Which bit(s) need to change to get it working?

140611 7:43:46 InnoDB: Operating system error number 25 in a file operation. InnoDB: Error number 25 means 'Inappropriate ioctl for device'. InnoDB: Some operating system error numbers are described at InnoDB:

http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html

...

...

InnoDB: File name ./ibdata1 InnoDB: File operation call: 'ioctl'. InnoDB: Error in opening ./ibdata1

-- -- Daniel Black, Engineer @ Open Query ( http://openquery.com.au ) Remote expertise & maintenance for MySQL/MariaDB server environments.

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- Mark Callaghan mdcallag@gmail.com

-- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

-- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

This e-mail (and any attachments) is confidential and may be privileged. Any unauthorized use, copying, disclosure or dissemination of this communication is prohibited. If you are not the intended recipient, please notify the sender immediately and delete all copies of the message and its attachments.