diff options
Diffstat (limited to 'Documentation/core-api')
| -rw-r--r-- | Documentation/core-api/bus-virt-phys-mapping.rst | 220 | ||||
| -rw-r--r-- | Documentation/core-api/cpu_hotplug.rst | 4 | ||||
| -rw-r--r-- | Documentation/core-api/dma-api.rst | 6 | ||||
| -rw-r--r-- | Documentation/core-api/dma-isa-lpc.rst | 2 | ||||
| -rw-r--r-- | Documentation/core-api/index.rst | 3 | ||||
| -rw-r--r-- | Documentation/core-api/kobject.rst | 2 | ||||
| -rw-r--r-- | Documentation/core-api/memory-allocation.rst | 44 | ||||
| -rw-r--r-- | Documentation/core-api/padata.rst | 18 | ||||
| -rw-r--r-- | Documentation/core-api/printk-basics.rst | 2 | ||||
| -rw-r--r-- | Documentation/core-api/printk-formats.rst | 8 | ||||
| -rw-r--r-- | Documentation/core-api/this_cpu_ops.rst | 339 | ||||
| -rw-r--r-- | Documentation/core-api/unaligned-memory-access.rst | 265 |
12 files changed, 887 insertions, 26 deletions
diff --git a/Documentation/core-api/bus-virt-phys-mapping.rst b/Documentation/core-api/bus-virt-phys-mapping.rst new file mode 100644 index 000000000000..c7bc99cd2e21 --- /dev/null +++ b/Documentation/core-api/bus-virt-phys-mapping.rst @@ -0,0 +1,220 @@ +========================================================== +How to access I/O mapped memory from within device drivers +========================================================== + +:Author: Linus + +.. warning:: + + The virt_to_bus() and bus_to_virt() functions have been + superseded by the functionality provided by the PCI DMA interface + (see :doc:`/core-api/dma-api-howto`). They continue + to be documented below for historical purposes, but new code + must not use them. --davidm 00/12/12 + +:: + + [ This is a mail message in response to a query on IO mapping, thus the + strange format for a "document" ] + +The AHA-1542 is a bus-master device, and your patch makes the driver give the +controller the physical address of the buffers, which is correct on x86 +(because all bus master devices see the physical memory mappings directly). + +However, on many setups, there are actually **three** different ways of looking +at memory addresses, and in this case we actually want the third, the +so-called "bus address". + +Essentially, the three ways of addressing memory are (this is "real memory", +that is, normal RAM--see later about other details): + + - CPU untranslated. This is the "physical" address. Physical address + 0 is what the CPU sees when it drives zeroes on the memory bus. + + - CPU translated address. This is the "virtual" address, and is + completely internal to the CPU itself with the CPU doing the appropriate + translations into "CPU untranslated". + + - bus address. This is the address of memory as seen by OTHER devices, + not the CPU. Now, in theory there could be many different bus + addresses, with each device seeing memory in some device-specific way, but + happily most hardware designers aren't actually actively trying to make + things any more complex than necessary, so you can assume that all + external hardware sees the memory the same way. + +Now, on normal PCs the bus address is exactly the same as the physical +address, and things are very simple indeed. However, they are that simple +because the memory and the devices share the same address space, and that is +not generally necessarily true on other PCI/ISA setups. + +Now, just as an example, on the PReP (PowerPC Reference Platform), the +CPU sees a memory map something like this (this is from memory):: + + 0-2 GB "real memory" + 2 GB-3 GB "system IO" (inb/out and similar accesses on x86) + 3 GB-4 GB "IO memory" (shared memory over the IO bus) + +Now, that looks simple enough. However, when you look at the same thing from +the viewpoint of the devices, you have the reverse, and the physical memory +address 0 actually shows up as address 2 GB for any IO master. + +So when the CPU wants any bus master to write to physical memory 0, it +has to give the master address 0x80000000 as the memory address. + +So, for example, depending on how the kernel is actually mapped on the +PPC, you can end up with a setup like this:: + + physical address: 0 + virtual address: 0xC0000000 + bus address: 0x80000000 + +where all the addresses actually point to the same thing. It's just seen +through different translations.. + +Similarly, on the Alpha, the normal translation is:: + + physical address: 0 + virtual address: 0xfffffc0000000000 + bus address: 0x40000000 + +(but there are also Alphas where the physical address and the bus address +are the same). + +Anyway, the way to look up all these translations, you do:: + + #include <asm/io.h> + + phys_addr = virt_to_phys(virt_addr); + virt_addr = phys_to_virt(phys_addr); + bus_addr = virt_to_bus(virt_addr); + virt_addr = bus_to_virt(bus_addr); + +Now, when do you need these? + +You want the **virtual** address when you are actually going to access that +pointer from the kernel. So you can have something like this:: + + /* + * this is the hardware "mailbox" we use to communicate with + * the controller. The controller sees this directly. + */ + struct mailbox { + __u32 status; + __u32 bufstart; + __u32 buflen; + .. + } mbox; + + unsigned char * retbuffer; + + /* get the address from the controller */ + retbuffer = bus_to_virt(mbox.bufstart); + switch (retbuffer[0]) { + case STATUS_OK: + ... + +on the other hand, you want the bus address when you have a buffer that +you want to give to the controller:: + + /* ask the controller to read the sense status into "sense_buffer" */ + mbox.bufstart = virt_to_bus(&sense_buffer); + mbox.buflen = sizeof(sense_buffer); + mbox.status = 0; + notify_controller(&mbox); + +And you generally **never** want to use the physical address, because you can't +use that from the CPU (the CPU only uses translated virtual addresses), and +you can't use it from the bus master. + +So why do we care about the physical address at all? We do need the physical +address in some cases, it's just not very often in normal code. The physical +address is needed if you use memory mappings, for example, because the +"remap_pfn_range()" mm function wants the physical address of the memory to +be remapped as measured in units of pages, a.k.a. the pfn (the memory +management layer doesn't know about devices outside the CPU, so it +shouldn't need to know about "bus addresses" etc). + +.. note:: + + The above is only one part of the whole equation. The above + only talks about "real memory", that is, CPU memory (RAM). + +There is a completely different type of memory too, and that's the "shared +memory" on the PCI or ISA bus. That's generally not RAM (although in the case +of a video graphics card it can be normal DRAM that is just used for a frame +buffer), but can be things like a packet buffer in a network card etc. + +This memory is called "PCI memory" or "shared memory" or "IO memory" or +whatever, and there is only one way to access it: the readb/writeb and +related functions. You should never take the address of such memory, because +there is really nothing you can do with such an address: it's not +conceptually in the same memory space as "real memory" at all, so you cannot +just dereference a pointer. (Sadly, on x86 it **is** in the same memory space, +so on x86 it actually works to just deference a pointer, but it's not +portable). + +For such memory, you can do things like: + + - reading:: + + /* + * read first 32 bits from ISA memory at 0xC0000, aka + * C000:0000 in DOS terms + */ + unsigned int signature = isa_readl(0xC0000); + + - remapping and writing:: + + /* + * remap framebuffer PCI memory area at 0xFC000000, + * size 1MB, so that we can access it: We can directly + * access only the 640k-1MB area, so anything else + * has to be remapped. + */ + void __iomem *baseptr = ioremap(0xFC000000, 1024*1024); + + /* write a 'A' to the offset 10 of the area */ + writeb('A',baseptr+10); + + /* unmap when we unload the driver */ + iounmap(baseptr); + + - copying and clearing:: + + /* get the 6-byte Ethernet address at ISA address E000:0040 */ + memcpy_fromio(kernel_buffer, 0xE0040, 6); + /* write a packet to the driver */ + memcpy_toio(0xE1000, skb->data, skb->len); + /* clear the frame buffer */ + memset_io(0xA0000, 0, 0x10000); + +OK, that just about covers the basics of accessing IO portably. Questions? +Comments? You may think that all the above is overly complex, but one day you +might find yourself with a 500 MHz Alpha in front of you, and then you'll be +happy that your driver works ;) + +Note that kernel versions 2.0.x (and earlier) mistakenly called the +ioremap() function "vremap()". ioremap() is the proper name, but I +didn't think straight when I wrote it originally. People who have to +support both can do something like:: + + /* support old naming silliness */ + #if LINUX_VERSION_CODE < 0x020100 + #define ioremap vremap + #define iounmap vfree + #endif + +at the top of their source files, and then they can use the right names +even on 2.0.x systems. + +And the above sounds worse than it really is. Most real drivers really +don't do all that complex things (or rather: the complexity is not so +much in the actual IO accesses as in error handling and timeouts etc). +It's generally not hard to fix drivers, and in many cases the code +actually looks better afterwards:: + + unsigned long signature = *(unsigned int *) 0xC0000; + vs + unsigned long signature = readl(0xC0000); + +I think the second version actually is more readable, no? diff --git a/Documentation/core-api/cpu_hotplug.rst b/Documentation/core-api/cpu_hotplug.rst index 4a50ab7817f7..f64668759b6a 100644 --- a/Documentation/core-api/cpu_hotplug.rst +++ b/Documentation/core-api/cpu_hotplug.rst @@ -35,8 +35,8 @@ Command Line Switches other CPUs later online. ``nr_cpus=n`` - Restrict the total amount CPUs the kernel will support. If the number - supplied here is lower than the number of physically available CPUs than + Restrict the total amount of CPUs the kernel will support. If the number + supplied here is lower than the number of physically available CPUs, then those CPUs can not be brought online later. ``additional_cpus=n`` diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst index f41620439ef3..3b3abbbb4b9a 100644 --- a/Documentation/core-api/dma-api.rst +++ b/Documentation/core-api/dma-api.rst @@ -5,7 +5,7 @@ Dynamic DMA mapping using the generic device :Author: James E.J. Bottomley <James.Bottomley@HansenPartnership.com> This document describes the DMA API. For a more gentle introduction -of the API (and actual examples), see Documentation/DMA-API-HOWTO.txt. +of the API (and actual examples), see :doc:`/core-api/dma-api-howto`. This API is split into two pieces. Part I describes the basic API. Part II describes extensions for supporting non-consistent memory @@ -479,7 +479,7 @@ without the _attrs suffixes, except that they pass an optional dma_attrs. The interpretation of DMA attributes is architecture-specific, and -each attribute should be documented in Documentation/DMA-attributes.txt. +each attribute should be documented in :doc:`/core-api/dma-attributes`. If dma_attrs are 0, the semantics of each of these functions is identical to those of the corresponding function @@ -492,7 +492,7 @@ for DMA:: #include <linux/dma-mapping.h> /* DMA_ATTR_FOO should be defined in linux/dma-mapping.h and - * documented in Documentation/DMA-attributes.txt */ + * documented in Documentation/core-api/dma-attributes.rst */ ... unsigned long attr; diff --git a/Documentation/core-api/dma-isa-lpc.rst b/Documentation/core-api/dma-isa-lpc.rst index b1ec7b16c21f..e59a3d35a93d 100644 --- a/Documentation/core-api/dma-isa-lpc.rst +++ b/Documentation/core-api/dma-isa-lpc.rst @@ -17,7 +17,7 @@ To do ISA style DMA you need to include two headers:: #include <asm/dma.h> The first is the generic DMA API used to convert virtual addresses to -bus addresses (see Documentation/DMA-API.txt for details). +bus addresses (see :doc:`/core-api/dma-api` for details). The second contains the routines specific to ISA DMA transfers. Since this is not present on all platforms make sure you construct your diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst index 15ab86112627..69171b1799f2 100644 --- a/Documentation/core-api/index.rst +++ b/Documentation/core-api/index.rst @@ -39,6 +39,8 @@ Library functionality that is used throughout the kernel. rbtree generic-radix-tree packing + bus-virt-phys-mapping + this_cpu_ops timekeeping errseq @@ -82,6 +84,7 @@ more memory-management documentation in :doc:`/vm/index`. :maxdepth: 1 memory-allocation + unaligned-memory-access dma-api dma-api-howto dma-attributes diff --git a/Documentation/core-api/kobject.rst b/Documentation/core-api/kobject.rst index e93dc8cf52dd..2739f8b72575 100644 --- a/Documentation/core-api/kobject.rst +++ b/Documentation/core-api/kobject.rst @@ -6,7 +6,7 @@ Everything you never wanted to know about kobjects, ksets, and ktypes :Last updated: December 19, 2007 Based on an original article by Jon Corbet for lwn.net written October 1, -2003 and located at http://lwn.net/Articles/51437/ +2003 and located at https://lwn.net/Articles/51437/ Part of the difficulty in understanding the driver model - and the kobject abstraction upon which it is built - is that there is no obvious starting diff --git a/Documentation/core-api/memory-allocation.rst b/Documentation/core-api/memory-allocation.rst index 4aa82ddd01b8..4446a1ac36cc 100644 --- a/Documentation/core-api/memory-allocation.rst +++ b/Documentation/core-api/memory-allocation.rst @@ -84,6 +84,50 @@ driver for a device with such restrictions, avoid using these flags. And even with hardware with restrictions it is preferable to use `dma_alloc*` APIs. +GFP flags and reclaim behavior +------------------------------ +Memory allocations may trigger direct or background reclaim and it is +useful to understand how hard the page allocator will try to satisfy that +or another request. + + * ``GFP_KERNEL & ~__GFP_RECLAIM`` - optimistic allocation without _any_ + attempt to free memory at all. The most light weight mode which even + doesn't kick the background reclaim. Should be used carefully because it + might deplete the memory and the next user might hit the more aggressive + reclaim. + + * ``GFP_KERNEL & ~__GFP_DIRECT_RECLAIM`` (or ``GFP_NOWAIT``)- optimistic + allocation without any attempt to free memory from the current + context but can wake kswapd to reclaim memory if the zone is below + the low watermark. Can be used from either atomic contexts or when + the request is a performance optimization and there is another + fallback for a slow path. + + * ``(GFP_KERNEL|__GFP_HIGH) & ~__GFP_DIRECT_RECLAIM`` (aka ``GFP_ATOMIC``) - + non sleeping allocation with an expensive fallback so it can access + some portion of memory reserves. Usually used from interrupt/bottom-half + context with an expensive slow path fallback. + + * ``GFP_KERNEL`` - both background and direct reclaim are allowed and the + **default** page allocator behavior is used. That means that not costly + allocation requests are basically no-fail but there is no guarantee of + that behavior so failures have to be checked properly by callers + (e.g. OOM killer victim is allowed to fail currently). + + * ``GFP_KERNEL | __GFP_NORETRY`` - overrides the default allocator behavior + and all allocation requests fail early rather than cause disruptive + reclaim (one round of reclaim in this implementation). The OOM killer + is not invoked. + + * ``GFP_KERNEL | __GFP_RETRY_MAYFAIL`` - overrides the default allocator + behavior and all allocation requests try really hard. The request + will fail if the reclaim cannot make any progress. The OOM killer + won't be triggered. + + * ``GFP_KERNEL | __GFP_NOFAIL`` - overrides the default allocator behavior + and all allocation requests will loop endlessly until they succeed. + This might be really dangerous especially for larger orders. + Selecting memory allocator ========================== diff --git a/Documentation/core-api/padata.rst b/Documentation/core-api/padata.rst index 0830e5b0e821..35175710b43c 100644 --- a/Documentation/core-api/padata.rst +++ b/Documentation/core-api/padata.rst @@ -27,22 +27,11 @@ padata_instance structure for overall control of how jobs are to be run:: #include <linux/padata.h> - struct padata_instance *padata_alloc_possible(const char *name); + struct padata_instance *padata_alloc(const char *name); 'name' simply identifies the instance. -There are functions for enabling and disabling the instance:: - - int padata_start(struct padata_instance *pinst); - void padata_stop(struct padata_instance *pinst); - -These functions are setting or clearing the "PADATA_INIT" flag; if that flag is -not set, other functions will refuse to work. padata_start() returns zero on -success (flag set) or -EINVAL if the padata cpumask contains no active CPU -(flag not set). padata_stop() clears the flag and blocks until the padata -instance is unused. - -Finally, complete padata initialization by allocating a padata_shell:: +Then, complete padata initialization by allocating a padata_shell:: struct padata_shell *padata_alloc_shell(struct padata_instance *pinst); @@ -155,11 +144,10 @@ submitted. Destroying ---------- -Cleaning up a padata instance predictably involves calling the three free +Cleaning up a padata instance predictably involves calling the two free functions that correspond to the allocation in reverse:: void padata_free_shell(struct padata_shell *ps); - void padata_stop(struct padata_instance *pinst); void padata_free(struct padata_instance *pinst); It is the user's responsibility to ensure all outstanding jobs are complete diff --git a/Documentation/core-api/printk-basics.rst b/Documentation/core-api/printk-basics.rst index 563a9ce5fe1d..965e4281eddd 100644 --- a/Documentation/core-api/printk-basics.rst +++ b/Documentation/core-api/printk-basics.rst @@ -69,7 +69,7 @@ You can check the current *console_loglevel* with:: The result shows the *current*, *default*, *minimum* and *boot-time-default* log levels. -To change the current console_loglevel simply write the the desired level to +To change the current console_loglevel simply write the desired level to ``/proc/sys/kernel/printk``. For example, to print all messages to the console:: # echo 8 > /proc/sys/kernel/printk diff --git a/Documentation/core-api/printk-formats.rst b/Documentation/core-api/printk-formats.rst index 8c9aba262b1e..6d26c5c6ac48 100644 --- a/Documentation/core-api/printk-formats.rst +++ b/Documentation/core-api/printk-formats.rst @@ -317,7 +317,7 @@ colon-separators. Leading zeros are always used. The additional ``c`` specifier can be used with the ``I`` specifier to print a compressed IPv6 address as described by -http://tools.ietf.org/html/rfc5952 +https://tools.ietf.org/html/rfc5952 Passed by reference. @@ -341,7 +341,7 @@ The additional ``p``, ``f``, and ``s`` specifiers are used to specify port flowinfo a ``/`` and scope a ``%``, each followed by the actual value. In case of an IPv6 address the compressed IPv6 address as described by -http://tools.ietf.org/html/rfc5952 is being used if the additional +https://tools.ietf.org/html/rfc5952 is being used if the additional specifier ``c`` is given. The IPv6 address is surrounded by ``[``, ``]`` in case of additional specifiers ``p``, ``f`` or ``s`` as suggested by https://tools.ietf.org/html/draft-ietf-6man-text-addr-representation-07 @@ -494,9 +494,11 @@ Time and date %pt[RT]t HH:MM:SS %pt[RT][dt][r] -For printing date and time as represented by +For printing date and time as represented by:: + R struct rtc_time structure T time64_t type + in human readable format. By default year will be incremented by 1900 and month by 1. diff --git a/Documentation/core-api/this_cpu_ops.rst b/Documentation/core-api/this_cpu_ops.rst new file mode 100644 index 000000000000..5cb8b883ae83 --- /dev/null +++ b/Documentation/core-api/this_cpu_ops.rst @@ -0,0 +1,339 @@ +=================== +this_cpu operations +=================== + +:Author: Christoph Lameter, August 4th, 2014 +:Author: Pranith Kumar, Aug 2nd, 2014 + +this_cpu operations are a way of optimizing access to per cpu +variables associated with the *currently* executing processor. This is +done through the use of segment registers (or a dedicated register where +the cpu permanently stored the beginning of the per cpu area for a +specific processor). + +this_cpu operations add a per cpu variable offset to the processor +specific per cpu base and encode that operation in the instruction +operating on the per cpu variable. + +This means that there are no atomicity issues between the calculation of +the offset and the operation on the data. Therefore it is not +necessary to disable preemption or interrupts to ensure that the +processor is not changed between the calculation of the address and +the operation on the data. + +Read-modify-write operations are of particular interest. Frequently +processors have special lower latency instructions that can operate +without the typical synchronization overhead, but still provide some +sort of relaxed atomicity guarantees. The x86, for example, can execute +RMW (Read Modify Write) instructions like inc/dec/cmpxchg without the +lock prefix and the associated latency penalty. + +Access to the variable without the lock prefix is not synchronized but +synchronization is not necessary since we are dealing with per cpu +data specific to the currently executing processor. Only the current +processor should be accessing that variable and therefore there are no +concurrency issues with other processors in the system. + +Please note that accesses by remote processors to a per cpu area are +exceptional situations and may impact performance and/or correctness +(remote write operations) of local RMW operations via this_cpu_*. + +The main use of the this_cpu operations has been to optimize counter +operations. + +The following this_cpu() operations with implied preemption protection +are defined. These operations can be used without worrying about +preemption and interrupts:: + + this_cpu_read(pcp) + this_cpu_write(pcp, val) + this_cpu_add(pcp, val) + this_cpu_and(pcp, val) + this_cpu_or(pcp, val) + this_cpu_add_return(pcp, val) + this_cpu_xchg(pcp, nval) + this_cpu_cmpxchg(pcp, oval, nval) + this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) + this_cpu_sub(pcp, val) + this_cpu_inc(pcp) + this_cpu_dec(pcp) + this_cpu_sub_return(pcp, val) + this_cpu_inc_return(pcp) + this_cpu_dec_return(pcp) + + +Inner working of this_cpu operations +------------------------------------ + +On x86 the fs: or the gs: segment registers contain the base of the +per cpu area. It is then possible to simply use the segment override +to relocate a per cpu relative address to the proper per cpu area for +the processor. So the relocation to the per cpu base is encoded in the +instruction via a segment register prefix. + +For example:: + + DEFINE_PER_CPU(int, x); + int z; + + z = this_cpu_read(x); + +results in a single instruction:: + + mov ax, gs:[x] + +instead of a sequence of calculation of the address and then a fetch +from that address which occurs with the per cpu operations. Before +this_cpu_ops such sequence also required preempt disable/enable to +prevent the kernel from moving the thread to a different processor +while the calculation is performed. + +Consider the following this_cpu operation:: + + this_cpu_inc(x) + +The above results in the following single instruction (no lock prefix!):: + + inc gs:[x] + +instead of the following operations required if there is no segment +register:: + + int *y; + int cpu; + + cpu = get_cpu(); + y = per_cpu_ptr(&x, cpu); + (*y)++; + put_cpu(); + +Note that these operations can only be used on per cpu data that is +reserved for a specific processor. Without disabling preemption in the +surrounding code this_cpu_inc() will only guarantee that one of the +per cpu counters is correctly incremented. However, there is no +guarantee that the OS will not move the process directly before or +after the this_cpu instruction is executed. In general this means that +the value of the individual counters for each processor are +meaningless. The sum of all the per cpu counters is the only value +that is of interest. + +Per cpu variables are used for performance reasons. Bouncing cache +lines can be avoided if multiple processors concurrently go through +the same code paths. Since each processor has its own per cpu +variables no concurrent cache line updates take place. The price that +has to be paid for this optimization is the need to add up the per cpu +counters when the value of a counter is needed. + + +Special operations +------------------ + +:: + + y = this_cpu_ptr(&x) + +Takes the offset of a per cpu variable (&x !) and returns the address +of the per cpu variable that belongs to the currently executing +processor. this_cpu_ptr avoids multiple steps that the common +get_cpu/put_cpu sequence requires. No processor number is +available. Instead, the offset of the local per cpu area is simply +added to the per cpu offset. + +Note that this operation is usually used in a code segment when +preemption has been disabled. The pointer is then used to +access local per cpu data in a critical section. When preemption +is re-enabled this pointer is usually no longer useful since it may +no longer point to per cpu data of the current processor. + + +Per cpu variables and offsets +----------------------------- + +Per cpu variables have *offsets* to the beginning of the per cpu +area. They do not have addresses although they look like that in the +code. Offsets cannot be directly dereferenced. The offset must be +added to a base pointer of a per cpu area of a processor in order to +form a valid address. + +Therefore the use of x or &x outside of the context of per cpu +operations is invalid and will generally be treated like a NULL +pointer dereference. + +:: + + DEFINE_PER_CPU(int, x); + +In the context of per cpu operations the above implies that x is a per +cpu variable. Most this_cpu operations take a cpu variable. + +:: + + int __percpu *p = &x; + +&x and hence p is the *offset* of a per cpu variable. this_cpu_ptr() +takes the offset of a per cpu variable which makes this look a bit +strange. + + +Operations on a field of a per cpu structure +-------------------------------------------- + +Let's say we have a percpu structure:: + + struct s { + int n,m; + }; + + DEFINE_PER_CPU(struct s, p); + + +Operations on these fields are straightforward:: + + this_cpu_inc(p.m) + + z = this_cpu_cmpxchg(p.m, 0, 1); + + +If we have an offset to struct s:: + + struct s __percpu *ps = &p; + + this_cpu_dec(ps->m); + + z = this_cpu_inc_return(ps->n); + + +The calculation of the pointer may require the use of this_cpu_ptr() +if we do not make use of this_cpu ops later to manipulate fields:: + + struct s *pp; + + pp = this_cpu_ptr(&p); + + pp->m--; + + z = pp->n++; + + +Variants of this_cpu ops +------------------------ + +this_cpu ops are interrupt safe. Some architectures do not support +these per cpu local operations. In that case the operation must be +replaced by code that disables interrupts, then does the operations +that are guaranteed to be atomic and then re-enable interrupts. Doing +so is expensive. If there are other reasons why the scheduler cannot +change the processor we are executing on then there is no reason to +disable interrupts. For that purpose the following __this_cpu operations +are provided. + +These operations have no guarantee against concurrent interrupts or +preemption. If a per cpu variable is not used in an interrupt context +and the scheduler cannot preempt, then they are safe. If any interrupts +still occur while an operation is in progress and if the interrupt too +modifies the variable, then RMW actions can not be guaranteed to be +safe:: + + __this_cpu_read(pcp) + __this_cpu_write(pcp, val) + __this_cpu_add(pcp, val) + __this_cpu_and(pcp, val) + __this_cpu_or(pcp, val) + __this_cpu_add_return(pcp, val) + __this_cpu_xchg(pcp, nval) + __this_cpu_cmpxchg(pcp, oval, nval) + __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) + __this_cpu_sub(pcp, val) + __this_cpu_inc(pcp) + __this_cpu_dec(pcp) + __this_cpu_sub_return(pcp, val) + __this_cpu_inc_return(pcp) + __this_cpu_dec_return(pcp) + + +Will increment x and will not fall-back to code that disables +interrupts on platforms that cannot accomplish atomicity through +address relocation and a Read-Modify-Write operation in the same +instruction. + + +&this_cpu_ptr(pp)->n vs this_cpu_ptr(&pp->n) +-------------------------------------------- + +The first operation takes the offset and forms an address and then +adds the offset of the n field. This may result in two add +instructions emitted by the compiler. + +The second one first adds the two offsets and then does the +relocation. IMHO the second form looks cleaner and has an easier time +with (). The second form also is consistent with the way +this_cpu_read() and friends are used. + + +Remote access to per cpu data +------------------------------ + +Per cpu data structures are designed to be used by one cpu exclusively. +If you use the variables as intended, this_cpu_ops() are guaranteed to +be "atomic" as no other CPU has access to these data structures. + +There are special cases where you might need to access per cpu data +structures remotely. It is usually safe to do a remote read access +and that is frequently done to summarize counters. Remote write access +something which could be problematic because this_cpu ops do not +have lock semantics. A remote write may interfere with a this_cpu +RMW operation. + +Remote write accesses to percpu data structures are highly discouraged +unless absolutely necessary. Please consider using an IPI to wake up +the remote CPU and perform the update to its per cpu area. + +To access per-cpu data structure remotely, typically the per_cpu_ptr() +function is used:: + + + DEFINE_PER_CPU(struct data, datap); + + struct data *p = per_cpu_ptr(&datap, cpu); + +This makes it explicit that we are getting ready to access a percpu +area remotely. + +You can also do the following to convert the datap offset to an address:: + + struct data *p = this_cpu_ptr(&datap); + +but, passing of pointers calculated via this_cpu_ptr to other cpus is +unusual and should be avoided. + +Remote access are typically only for reading the status of another cpus +per cpu data. Write accesses can cause unique problems due to the +relaxed synchronization requirements for this_cpu operations. + +One example that illustrates some concerns with write operations is +the following scenario that occurs because two per cpu variables +share a cache-line but the relaxed synchronization is applied to +only one process updating the cache-line. + +Consider the following example:: + + + struct test { + atomic_t a; + int b; + }; + + DEFINE_PER_CPU(struct test, onecacheline); + +There is some concern about what would happen if the field 'a' is updated +remotely from one processor and the local processor would use this_cpu ops +to update field b. Care should be taken that such simultaneous accesses to +data within the same cache line are avoided. Also costly synchronization +may be necessary. IPIs are generally recommended in such scenarios instead +of a remote write to the per cpu area of another processor. + +Even in cases where the remote writes are rare, please bear in +mind that a remote write will evict the cache line from the processor +that most likely will access it. If the processor wakes up and finds a +missing local cache line of a per cpu area, its performance and hence +the wake up times will be affected. diff --git a/Documentation/core-api/unaligned-memory-access.rst b/Documentation/core-api/unaligned-memory-access.rst new file mode 100644 index 000000000000..1ee82419d8aa --- /dev/null +++ b/Documentation/core-api/unaligned-memory-access.rst @@ -0,0 +1,265 @@ +========================= +Unaligned Memory Accesses +========================= + +:Author: Daniel Drake <dsd@gentoo.org>, +:Author: Johannes Berg <johannes@sipsolutions.net> + +:With help from: Alan Cox, Avuton Olrich, Heikki Orsila, Jan Engelhardt, + Kyle McMartin, Kyle Moffett, Randy Dunlap, Robert Hancock, Uli Kunitz, + Vadim Lobanov + + +Linux runs on a wide variety of architectures which have varying behaviour +when it comes to memory access. This document presents some details about +unaligned accesses, why you need to write code that doesn't cause them, +and how to write such code! + + +The definition of an unaligned access +===================================== + +Unaligned memory accesses occur when you try to read N bytes of data starting +from an address that is not evenly divisible by N (i.e. addr % N != 0). +For example, reading 4 bytes of data from address 0x10004 is fine, but +reading 4 bytes of data from address 0x10005 would be an unaligned memory +access. + +The above may seem a little vague, as memory access can happen in different +ways. The context here is at the machine code level: certain instructions read +or write a number of bytes to or from memory (e.g. movb, movw, movl in x86 +assembly). As will become clear, it is relatively easy to spot C statements +which will compile to multiple-byte memory access instructions, namely when +dealing with types such as u16, u32 and u64. + + +Natural alignment +================= + +The rule mentioned above forms what we refer to as natural alignment: +When accessing N bytes of memory, the base memory address must be evenly +divisible by N, i.e. addr % N == 0. + +When writing code, assume the target architecture has natural alignment +requirements. + +In reality, only a few architectures require natural alignment on all sizes +of memory access. However, we must consider ALL supported architectures; +writing code that satisfies natural alignment requirements is the easiest way +to achieve full portability. + + +Why unaligned access is bad +=========================== + +The effects of performing an unaligned memory access vary from architecture +to architecture. It would be easy to write a whole document on the differences +here; a summary of the common scenarios is presented below: + + - Some architectures are able to perform unaligned memory accesses + transparently, but there is usually a significant performance cost. + - Some architectures raise processor exceptions when unaligned accesses + happen. The exception handler is able to correct the unaligned access, + at significant cost to performance. + - Some architectures raise processor exceptions when unaligned accesses + happen, but the exceptions do not contain enough information for the + unaligned access to be corrected. + - Some architectures are not capable of unaligned memory access, but will + silently perform a different memory access to the one that was requested, + resulting in a subtle code bug that is hard to detect! + +It should be obvious from the above that if your code causes unaligned +memory accesses to happen, your code will not work correctly on certain +platforms and will cause performance problems on others. + + +Code that does not cause unaligned access +========================================= + +At first, the concepts above may seem a little hard to relate to actual +coding practice. After all, you don't have a great deal of control over +memory addresses of certain variables, etc. + +Fortunately things are not too complex, as in most cases, the compiler +ensures that things will work for you. For example, take the following +structure:: + + struct foo { + u16 field1; + u32 field2; + u8 field3; + }; + +Let us assume that an instance of the above structure resides in memory +starting at address 0x10000. With a basic level of understanding, it would +not be unreasonable to expect that accessing field2 would cause an unaligned +access. You'd be expecting field2 to be located at offset 2 bytes into the +structure, i.e. address 0x10002, but that address is not evenly divisible +by 4 (remember, we're reading a 4 byte value here). + +Fortunately, the compiler understands the alignment constraints, so in the +above case it would insert 2 bytes of padding in between field1 and field2. +Therefore, for standard structure types you can always rely on the compiler +to pad structures so that accesses to fields are suitably aligned (assuming +you do not cast the field to a type of different length). + +Similarly, you can also rely on the compiler to align variables and function +parameters to a naturally aligned scheme, based on the size of the type of +the variable. + +At this point, it should be clear that accessing a single byte (u8 or char) +will never cause an unaligned access, because all memory addresses are evenly +divisible by one. + +On a related topic, with the above considerations in mind you may observe +that you could reorder the fields in the structure in order to place fields +where padding would otherwise be inserted, and hence reduce the overall +resident memory size of structure instances. The optimal layout of the +above example is:: + + struct foo { + u32 field2; + u16 field1; + u8 field3; + }; + +For a natural alignment scheme, the compiler would only have to add a single +byte of padding at the end of the structure. This padding is added in order +to satisfy alignment constraints for arrays of these structures. + +Another point worth mentioning is the use of __attribute__((packed)) on a +structure type. This GCC-specific attribute tells the compiler never to +insert any padding within structures, useful when you want to use a C struct +to represent some data that comes in a fixed arrangement 'off the wire'. + +You might be inclined to believe that usage of this attribute can easily +lead to unaligned accesses when accessing fields that do not satisfy +architectural alignment requirements. However, again, the compiler is aware +of the alignment constraints and will generate extra instructions to perform +the memory access in a way that does not cause unaligned access. Of course, +the extra instructions obviously cause a loss in performance compared to the +non-packed case, so the packed attribute should only be used when avoiding +structure padding is of importance. + + +Code that causes unaligned access +================================= + +With the above in mind, let's move onto a real life example of a function +that can cause an unaligned memory access. The following function taken +from include/linux/etherdevice.h is an optimized routine to compare two +ethernet MAC addresses for equality:: + + bool ether_addr_equal(const u8 *addr1, const u8 *addr2) + { + #ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS + u32 fold = ((*(const u32 *)addr1) ^ (*(const u32 *)addr2)) | + ((*(const u16 *)(addr1 + 4)) ^ (*(const u16 *)(addr2 + 4))); + + return fold == 0; + #else + const u16 *a = (const u16 *)addr1; + const u16 *b = (const u16 *)addr2; + return ((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2])) == 0; + #endif + } + +In the above function, when the hardware has efficient unaligned access +capability, there is no issue with this code. But when the hardware isn't +able to access memory on arbitrary boundaries, the reference to a[0] causes +2 bytes (16 bits) to be read from memory starting at address addr1. + +Think about what would happen if addr1 was an odd address such as 0x10003. +(Hint: it'd be an unaligned access.) + +Despite the potential unaligned access problems with the above function, it +is included in the kernel anyway but is understood to only work normally on +16-bit-aligned addresses. It is up to the caller to ensure this alignment or +not use this function at all. This alignment-unsafe function is still useful +as it is a decent optimization for the cases when you can ensure alignment, +which is true almost all of the time in ethernet networking context. + + +Here is another example of some code that could cause unaligned accesses:: + + void myfunc(u8 *data, u32 value) + { + [...] + *((u32 *) data) = cpu_to_le32(value); + [...] + } + +This code will cause unaligned accesses every time the data parameter points +to an address that is not evenly divisible by 4. + +In summary, the 2 main scenarios where you may run into unaligned access +problems involve: + + 1. Casting variables to types of different lengths + 2. Pointer arithmetic followed by access to at least 2 bytes of data + + +Avoiding unaligned accesses +=========================== + +The easiest way to avoid unaligned access is to use the get_unaligned() and +put_unaligned() macros provided by the <asm/unaligned.h> header file. + +Going back to an earlier example of code that potentially causes unaligned +access:: + + void myfunc(u8 *data, u32 value) + { + [...] + *((u32 *) data) = cpu_to_le32(value); + [...] + } + +To avoid the unaligned memory access, you would rewrite it as follows:: + + void myfunc(u8 *data, u32 value) + { + [...] + value = cpu_to_le32(value); + put_unaligned(value, (u32 *) data); + [...] + } + +The get_unaligned() macro works similarly. Assuming 'data' is a pointer to +memory and you wish to avoid unaligned access, its usage is as follows:: + + u32 value = get_unaligned((u32 *) data); + +These macros work for memory accesses of any length (not just 32 bits as +in the examples above). Be aware that when compared to standard access of +aligned memory, using these macros to access unaligned memory can be costly in +terms of performance. + +If use of such macros is not convenient, another option is to use memcpy(), +where the source or destination (or both) are of type u8* or unsigned char*. +Due to the byte-wise nature of this operation, unaligned accesses are avoided. + + +Alignment vs. Networking +======================== + +On architectures that require aligned loads, networking requires that the IP +header is aligned on a four-byte boundary to optimise the IP stack. For +regular ethernet hardware, the constant NET_IP_ALIGN is used. On most +architectures this constant has the value 2 because the normal ethernet +header is 14 bytes long, so in order to get proper alignment one needs to +DMA to an address which can be expressed as 4*n + 2. One notable exception +here is powerpc which defines NET_IP_ALIGN to 0 because DMA to unaligned +addresses can be very expensive and dwarf the cost of unaligned loads. + +For some ethernet hardware that cannot DMA to unaligned addresses like +4*n+2 or non-ethernet hardware, this can be a problem, and it is then +required to copy the incoming frame into an aligned buffer. Because this is +unnecessary on architectures that can do unaligned accesses, the code can be +made dependent on CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS like so:: + + #ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS + skb = original skb + #else + skb = copy skb + #endif |