diff options
297 files changed, 50658 insertions, 4 deletions
@@ -348,10 +348,22 @@ MigrationBackup/ # Ionide (cross platform F# VS Code tools) working folder .ionide/ -*/Taskfile.yml -*.yaml + +#Task build files +**/[Tt]askfile* + +#no build dlls in source *.dll -/Net.Http/src/Properties/launchSettings.json + +**[Ll]aunchSettings* + +#licenseheader vs extension files *.licenseheader -/WinRpMalloc/packages.config + +#package config files +**[Pp]ackages* + +#Do not include sln files for build process only *.sln + +*.filters
\ No newline at end of file diff --git a/Hashing.Portable/LICENSE.txt b/Hashing.Portable/LICENSE.txt new file mode 100644 index 0000000..895db28 --- /dev/null +++ b/Hashing.Portable/LICENSE.txt @@ -0,0 +1,293 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). + +SPDX-License-Identifier: GPL-2.0-or-later + +License-Text: + +GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Hashing.Portable/README.md b/Hashing.Portable/README.md new file mode 100644 index 0000000..5ebf6be --- /dev/null +++ b/Hashing.Portable/README.md @@ -0,0 +1,86 @@ + +# VNLib.Hashing.Portable + +This library is a collection of common cryptographic functions, optimized using the VNLib.Utils +library for interop and memory management. + +## Argon2 +This library contains an native library interface with the Argon2 Cryptographic Hashing library. If you wish to use the Argon2 hashing functions, you must include the [Argon2 native library](https://github.com/P-H-C/phc-winner-argon2) in your project, and accept the license. + +The Argon2 native libary is lazy loaded and therefor not required for the other functions in this library, if it is not included. You may specify the exact path to the native library by setting the `ARGON2_DLL_PATH`environment variable to the value of the path. + +**Notice:** +This library does not, modify, contribute, or affect the functionality of the Argon2 library in any way. + +#### Usage: +``` +//Using the managed hash version, inputs may be binary or utf8 chars +string encodedHash = VnArgon2.Hash2id(<password>,<salt>,<secret>,...<argon params>) + +//The 'raw' or 'passthru' 2id managed hashing method, binary only +VnArgon2.Hash2id(<passbytes>,<saltbytes><secretbytes>,<rawHashOutput>,...<params>) + +//Verification used CryptographicOperations.FixedTimeEquals for comparison +//managed verification, only valid with previously hashed methods +bool valid = VnArgon2.Verify2id(<rawPass>,<hash>,<encodedHash>) + +//Binary only 'raw' or 'passthru' 2id managed verification +bool valid = VnArgon2.Verify2id(<rawPass>,<salt>,<secret>,<rawHashBytes>) +``` + +## Other Classes + +The ManagedHash and RandomHash classes are simple "shortcut" methods for common hashing operations with common data encoding/decoding. + +The IdentityUtility namespace includes classes and methods for generating and validating JWE types, such as JWT (Json Web Token) and JWK (Json Web Key), and their various extension/helper methods. + +### Basic Usage +``` +//RandomHash +byte[] cngBytes = RandomHash.GetRandomBytes(); +RandomHash.GetRandomBytes(<binary span>); +string base64 = RandomHash.GetRandomBase64(<size>); +string base32 = RandomHash.GetRandomBase32(<size>); +string hex = RandomHash.GetRandomHex(<size>); +string encodedHash = RandomHash.GetRandomHash(<hashAlg>,<size>,<encoding>); +GUID cngGuid = RandomHash.GetSecureGuid(); + +//Managed hash +ERRNO result = ManagedHash.ComputeHash(<data>,<args>); +string encoded = ManagedHash.ComputeHash(<data>,<args>); +byte[] rawHash = ManagedHash.ComputeHash(<data>,<args>); + +//HMAC +ERRNO result = ManagedHash.ComputeHmac(<key>,<data>,<args>); +string encoded = ManagedHash.ComputeHmac(<key>,<data>,<args>); +byte[] rawHash = ManagedHash.ComputeHmac(<key>,<data>,<args>); + + +//Parse jwt +using JsonWebToken jwt = JsonWebToken.Parse(<jwtEncodedString>); +bool valid = jwt.verify(<Algorithm>,<hashMethod>...); +//Get the payload (or header, they use the same methods) +T payload = jwt.GetPaylod<T>();//OR +JsonDocument payload = jwt.GetPayload(); + +//Create new JWT +using JsonWebToken jwt = new(<optionalHeap>); +jwt.WriteHeader(<object or binary>); //Set header + +jwt.WritePayload(<object or binary>); //Set by serializing it, or binary + +//OR init fluent payload builder +jwt.InitPayloadClaim() + .AddClaim(<string name>, <object value>) + ... + .CommitClaims(); //Serializes the claims and writes them to the JWT payload + +jwt.Sign(<HashAlgorithm, RSA, ECDsa>... <params>); //Sign the JWT + +string jwtData = jwt.Compile(); //Serialize the JWT +``` + +### License + +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). +See the LICENSE files for more information.
\ No newline at end of file diff --git a/Hashing.Portable/src/Argon2/Argon2PasswordEntry.cs b/Hashing.Portable/src/Argon2/Argon2PasswordEntry.cs new file mode 100644 index 0000000..bc57d7a --- /dev/null +++ b/Hashing.Portable/src/Argon2/Argon2PasswordEntry.cs @@ -0,0 +1,117 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: Argon2PasswordEntry.cs +* +* Argon2PasswordEntry.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +/* + * VnArgon2.cs + * Author: Vaughhn Nugent + * Date: July 17, 2021 + * + * Dependencies Argon2. + * https://github.com/P-H-C/phc-winner-argon2 + * + */ + +using System; +using System.Globalization; + +using VNLib.Utils.Extensions; + +namespace VNLib.Hashing +{ + + public static unsafe partial class VnArgon2 + { + private readonly ref struct Argon2PasswordEntry + { + public readonly uint TimeCost; + public readonly uint MemoryCost; + public readonly Argon2_version Version; + public readonly uint Parallelism; + public readonly ReadOnlySpan<char> Salt; + public readonly ReadOnlySpan<char> Hash; + + private static Argon2_version ParseVersion(ReadOnlySpan<char> window) + { + //Version comes after the v= prefix + ReadOnlySpan<char> v = window.SliceAfterParam(",v="); + v = v.SliceBeforeParam(','); + //Parse the version as an enum value + return Enum.Parse<Argon2_version>(v); + } + + private static uint ParseTimeCost(ReadOnlySpan<char> window) + { + //TimeCost comes after the t= prefix + ReadOnlySpan<char> t = window.SliceAfterParam(",t="); + t = t.SliceBeforeParam(','); + //Parse the time cost as an unsigned integer + return uint.Parse(t, NumberStyles.Integer, CultureInfo.InvariantCulture); + } + + private static uint ParseMemoryCost(ReadOnlySpan<char> window) + { + //MemoryCost comes after the m= prefix + ReadOnlySpan<char> m = window.SliceAfterParam(",m="); + m = m.SliceBeforeParam(','); + //Parse the memory cost as an unsigned integer + return uint.Parse(m, NumberStyles.Integer, CultureInfo.InvariantCulture); + } + + private static uint ParseParallelism(ReadOnlySpan<char> window) + { + //Parallelism comes after the p= prefix + ReadOnlySpan<char> p = window.SliceAfterParam(",p="); + p = p.SliceBeforeParam(','); + //Parse the parallelism as an unsigned integer + return uint.Parse(p, NumberStyles.Integer, CultureInfo.InvariantCulture); + } + + private static ReadOnlySpan<char> ParseSalt(ReadOnlySpan<char> window) + { + //Salt comes after the s= prefix + ReadOnlySpan<char> s = window.SliceAfterParam(",s="); + s = s.SliceBeforeParam('$'); + //Parse the salt as a string + return s; + } + + private static ReadOnlySpan<char> ParseHash(ReadOnlySpan<char> window) + { + //Get last index of dollar sign for the start of the password hash + int start = window.LastIndexOf('$'); + return window[(start + 1)..]; + } + + public Argon2PasswordEntry(ReadOnlySpan<char> str) + { + Version = ParseVersion(str); + TimeCost = ParseTimeCost(str); + MemoryCost = ParseMemoryCost(str); + Parallelism = ParseParallelism(str); + Salt = ParseSalt(str); + Hash = ParseHash(str); + } + } + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/Argon2/Argon2_Context.cs b/Hashing.Portable/src/Argon2/Argon2_Context.cs new file mode 100644 index 0000000..78d0f13 --- /dev/null +++ b/Hashing.Portable/src/Argon2/Argon2_Context.cs @@ -0,0 +1,74 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: Argon2_Context.cs +* +* Argon2_Context.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +/* + * VnArgon2.cs + * Author: Vaughhn Nugent + * Date: July 17, 2021 + * + * Dependencies Argon2. + * https://github.com/P-H-C/phc-winner-argon2 + * + */ + +using System; +using System.Runtime.InteropServices; + +namespace VNLib.Hashing +{ + + public static unsafe partial class VnArgon2 + { + [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)] + private ref struct Argon2_Context + { + public void* outptr; /* output array */ + public UInt32 outlen; /* digest length */ + + public void* pwd; /* password array */ + public UInt32 pwdlen; /* password length */ + + public void* salt; /* salt array */ + public UInt32 saltlen; /* salt length */ + + public void* secret; /* key array */ + public UInt32 secretlen; /* key length */ + + public void* ad; /* associated data array */ + public UInt32 adlen; /* associated data length */ + + public UInt32 t_cost; /* number of passes */ + public UInt32 m_cost; /* amount of memory requested (KB) */ + public UInt32 lanes; /* number of lanes */ + public UInt32 threads; /* maximum number of threads */ + + public Argon2_version version; /* version number */ + + public void* allocate_cbk; /* pointer to memory allocator */ + public void* free_cbk; /* pointer to memory deallocator */ + + public UInt32 flags; /* array of bool options */ + } + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/Argon2/Argon2_ErrorCodes.cs b/Hashing.Portable/src/Argon2/Argon2_ErrorCodes.cs new file mode 100644 index 0000000..70a6764 --- /dev/null +++ b/Hashing.Portable/src/Argon2/Argon2_ErrorCodes.cs @@ -0,0 +1,76 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: Argon2_ErrorCodes.cs +* +* Argon2_ErrorCodes.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +/* + * VnArgon2.cs + * Author: Vaughhn Nugent + * Date: July 17, 2021 + * + * Dependencies Argon2. + * https://github.com/P-H-C/phc-winner-argon2 + * + */ + +namespace VNLib.Hashing +{ + public enum Argon2_ErrorCodes + { + ARGON2_OK = 0, + ARGON2_OUTPUT_PTR_NULL = -1, + ARGON2_OUTPUT_TOO_SHORT = -2, + ARGON2_OUTPUT_TOO_LONG = -3, + ARGON2_PWD_TOO_SHORT = -4, + ARGON2_PWD_TOO_LONG = -5, + ARGON2_SALT_TOO_SHORT = -6, + ARGON2_SALT_TOO_LONG = -7, + ARGON2_AD_TOO_SHORT = -8, + ARGON2_AD_TOO_LONG = -9, + ARGON2_SECRET_TOO_SHORT = -10, + ARGON2_SECRET_TOO_LONG = -11, + ARGON2_TIME_TOO_SMALL = -12, + ARGON2_TIME_TOO_LARGE = -13, + ARGON2_MEMORY_TOO_LITTLE = -14, + ARGON2_MEMORY_TOO_MUCH = -15, + ARGON2_LANES_TOO_FEW = -16, + ARGON2_LANES_TOO_MANY = -17, + ARGON2_PWD_PTR_MISMATCH = -18, /* NULL ptr with non-zero length */ + ARGON2_SALT_PTR_MISMATCH = -19, /* NULL ptr with non-zero length */ + ARGON2_SECRET_PTR_MISMATCH = -20, /* NULL ptr with non-zero length */ + ARGON2_AD_PTR_MISMATCH = -21, /* NULL ptr with non-zero length */ + ARGON2_MEMORY_ALLOCATION_ERROR = -22, + ARGON2_FREE_MEMORY_CBK_NULL = -23, + ARGON2_ALLOCATE_MEMORY_CBK_NULL = -24, + ARGON2_INCORRECT_PARAMETER = -25, + ARGON2_INCORRECT_TYPE = -26, + ARGON2_OUT_PTR_MISMATCH = -27, + ARGON2_THREADS_TOO_FEW = -28, + ARGON2_THREADS_TOO_MANY = -29, + ARGON2_MISSING_ARGS = -30, + ARGON2_ENCODING_FAIL = -31, + ARGON2_DECODING_FAIL = -32, + ARGON2_THREAD_FAIL = -33, + ARGON2_DECODING_LENGTH_FAIL = -34, + ARGON2_VERIFY_MISMATCH = -35 + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/Argon2/VnArgon2.cs b/Hashing.Portable/src/Argon2/VnArgon2.cs new file mode 100644 index 0000000..2e98ddb --- /dev/null +++ b/Hashing.Portable/src/Argon2/VnArgon2.cs @@ -0,0 +1,438 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: VnArgon2.cs +* +* VnArgon2.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + + +using System; +using System.Text; +using System.Threading; +using System.Buffers.Text; +using System.Security.Cryptography; + +using VNLib.Utils.Memory; +using VNLib.Utils.Native; +using VNLib.Utils.Extensions; + +namespace VNLib.Hashing +{ + + /// <summary> + /// Implements the Argon2 data hashing library in .NET for cross platform use. + /// </summary> + /// <remarks>Buffers are allocted on a private <see cref="IUnmangedHeap"/> instance.</remarks> + public static unsafe partial class VnArgon2 + { + public const uint ARGON2_DEFAULT_FLAGS = 0U; + public const uint HASH_SIZE = 128; + public const int MAX_SALT_SIZE = 100; + public const string ID_MODE = "argon2id"; + public const string ARGON2_CTX_SAFE_METHOD_NAME = "argon2id_ctx"; + public const string ARGON2_LIB_ENVIRONMENT_VAR_NAME = "ARGON2_DLL_PATH"; + public const string ARGON2_DEFUALT_LIB_NAME = "Argon2"; + + private static readonly Encoding LocEncoding = Encoding.Unicode; + private static readonly Lazy<IUnmangedHeap> _heap = new (Memory.InitializeNewHeapForProcess, LazyThreadSafetyMode.PublicationOnly); + private static readonly Lazy<Argon2NativeLibary> _nativeLibrary = new(LoadNativeLib, LazyThreadSafetyMode.PublicationOnly); + + + //Private heap initialized to 10k size, and all allocated buffers will be zeroed when allocated + private static IUnmangedHeap PwHeap => _heap.Value; + + /* Argon2 primitive type */ + private enum Argon2_type + { + Argon2_d, Argon2_i, Argon2_id + } + + /* Version of the algorithm */ + private enum Argon2_version + { + ARGON2_VERSION_10 = 0x10, + ARGON2_VERSION_13 = 0x13, + ARGON2_VERSION_NUMBER = ARGON2_VERSION_13 + } + + /* + * The native library delegate method + */ + delegate int Argon2InvokeHash(Argon2_Context* context); + + /* + * Wrapper class that manages lifetime of the native library + * This should basically never get finalized, but if it does + * it will free the native lib + */ + private class Argon2NativeLibary + { + private readonly SafeMethodHandle<Argon2InvokeHash> _argon2id_ctx; + + public Argon2NativeLibary(SafeMethodHandle<Argon2InvokeHash> method) => _argon2id_ctx = method; + + public int Argon2Hash(Argon2_Context* context) => _argon2id_ctx.Method!.Invoke(context); + + ~Argon2NativeLibary() + { + //Dispose method handle which will release the native library + _argon2id_ctx.Dispose(); + } + } + + /// <summary> + /// Loads the native Argon2 libray into the process with env variable library path + /// </summary> + /// <returns></returns> + private static Argon2NativeLibary LoadNativeLib() + { + //Get the path to the argon2 library + string? argon2EnvPath = Environment.GetEnvironmentVariable(ARGON2_LIB_ENVIRONMENT_VAR_NAME); + //Default to the default library name + argon2EnvPath ??= ARGON2_DEFUALT_LIB_NAME; + + //Try to load the libary and always dispose it so the native method handle will unload the library + using SafeLibraryHandle lib = SafeLibraryHandle.LoadLibrary(argon2EnvPath); + + //Get safe native method + SafeMethodHandle<Argon2InvokeHash> method = lib.GetMethod<Argon2InvokeHash>(ARGON2_CTX_SAFE_METHOD_NAME); + + return new Argon2NativeLibary(method); + } + + /// <summary> + /// Hashes a password with a salt and specified arguments + /// </summary> + /// <param name="password">Span of characters containing the password to be hashed</param> + /// <param name="salt">Span of characters contating the salt to include in the hashing</param> + /// <param name="secret">Optional secret to include in hash</param> + /// <param name="hashLen">Size of the hash in bytes</param> + /// <param name="memCost">Memory cost</param> + /// <param name="parallelism">Degree of parallelism</param> + /// <param name="timeCost">Time cost of operation</param> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="InsufficientMemoryException"></exception> + /// <returns>A <see cref="Encoding.Unicode"/> <see cref="string"/> containg the ready-to-store hash</returns> + public static string Hash2id(ReadOnlySpan<char> password, ReadOnlySpan<char> salt, ReadOnlySpan<byte> secret, + uint timeCost = 2, uint memCost = 65535, uint parallelism = 4, uint hashLen = HASH_SIZE) + { + //Get bytes count + int saltbytes = LocEncoding.GetByteCount(salt); + //Get bytes count for password + int passBytes = LocEncoding.GetByteCount(password); + //Alloc memory for salt + using MemoryHandle<byte> buffer = PwHeap.Alloc<byte>(saltbytes + passBytes, true); + Span<byte> saltBuffer = buffer.AsSpan(0, saltbytes); + Span<byte> passBuffer = buffer.AsSpan(passBytes); + //Encode salt with span the same size of the salt + _ = LocEncoding.GetBytes(salt, saltBuffer); + //Encode password, create a new span to make sure its proper size + _ = LocEncoding.GetBytes(password, passBuffer); + //Hash + return Hash2id(passBuffer, saltBuffer, secret, timeCost, memCost, parallelism, hashLen); + } + + /// <summary> + /// Hashes a password with a salt and specified arguments + /// </summary> + /// <param name="password">Span of characters containing the password to be hashed</param> + /// <param name="salt">Span of characters contating the salt to include in the hashing</param> + /// <param name="secret">Optional secret to include in hash</param> + /// <param name="hashLen">Size of the hash in bytes</param> + /// <param name="memCost">Memory cost</param> + /// <param name="parallelism">Degree of parallelism</param> + /// <param name="timeCost">Time cost of operation</param> + /// <exception cref="FormatException"></exception> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="InsufficientMemoryException"></exception> + /// <returns>A <see cref="Encoding.Unicode"/> <see cref="string"/> containg the ready-to-store hash</returns> + public static string Hash2id(ReadOnlySpan<char> password, ReadOnlySpan<byte> salt, ReadOnlySpan<byte> secret, + uint timeCost = 2, uint memCost = 65535, uint parallelism = 4, uint hashLen = HASH_SIZE) + { + //Get bytes count + int passBytes = LocEncoding.GetByteCount(password); + //Alloc memory for password + using MemoryHandle<byte> pwdHandle = PwHeap.Alloc<byte>(passBytes, true); + //Encode password, create a new span to make sure its proper size + _ = LocEncoding.GetBytes(password, pwdHandle); + //Hash + return Hash2id(pwdHandle.Span, salt, secret, timeCost, memCost, parallelism, hashLen); + } + + /// <summary> + /// Hashes a password with a salt and specified arguments + /// </summary> + /// <param name="password">Span of characters containing the password to be hashed</param> + /// <param name="salt">Span of characters contating the salt to include in the hashing</param> + /// <param name="secret">Optional secret to include in hash</param> + /// <param name="hashLen">Size of the hash in bytes</param> + /// <param name="memCost">Memory cost</param> + /// <param name="parallelism">Degree of parallelism</param> + /// <param name="timeCost">Time cost of operation</param> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <returns>A <see cref="Encoding.Unicode"/> <see cref="string"/>containg the ready-to-store hash</returns> + public static string Hash2id(ReadOnlySpan<byte> password, ReadOnlySpan<byte> salt, ReadOnlySpan<byte> secret, + uint timeCost = 2, uint memCost = 65535, uint parallelism = 4, uint hashLen = HASH_SIZE) + { + string hash, salts; + //Alloc data for hash output + using MemoryHandle<byte> hashHandle = PwHeap.Alloc<byte>(hashLen, true); + //hash the password + Hash2id(password, salt, secret, hashHandle.Span, timeCost, memCost, parallelism); + //Encode hash + hash = Convert.ToBase64String(hashHandle.Span); + //encode salt + salts = Convert.ToBase64String(salt); + //Encode salt in base64 + return $"${ID_MODE},v={(int)Argon2_version.ARGON2_VERSION_13},m={memCost},t={timeCost},p={parallelism},s={salts}${hash}"; + } + + /// <summary> + /// Exposes the raw Argon2-ID hashing api to C#, using spans (pins memory references) + /// </summary> + /// <param name="password">Span of characters containing the password to be hashed</param> + /// <param name="rawHashOutput">The output buffer to store the raw hash output</param> + /// <param name="salt">Span of characters contating the salt to include in the hashing</param> + /// <param name="secret">Optional secret to include in hash</param> + /// <param name="memCost">Memory cost</param> + /// <param name="parallelism">Degree of parallelism</param> + /// <param name="timeCost">Time cost of operation</param> + /// <exception cref="VnArgon2Exception"></exception> + public static void Hash2id(ReadOnlySpan<byte> password, ReadOnlySpan<byte> salt, ReadOnlySpan<byte> secret, in Span<byte> rawHashOutput, + uint timeCost = 2, uint memCost = 65535, uint parallelism = 4) + { + fixed (byte* pwd = password, slptr = salt, secretptr = secret, outPtr = rawHashOutput) + { + //Setup context + Argon2_Context ctx; + //Pointer + Argon2_Context* context = &ctx; + context->version = Argon2_version.ARGON2_VERSION_13; + context->t_cost = timeCost; + context->m_cost = memCost; + context->threads = parallelism; + context->lanes = parallelism; + //Default flags + context->flags = ARGON2_DEFAULT_FLAGS; + context->allocate_cbk = null; + context->free_cbk = null; + //Password + context->pwd = pwd; + context->pwdlen = (UInt32)password.Length; + //Salt + context->salt = slptr; + context->saltlen = (UInt32)salt.Length; + //Secret + context->secret = secretptr; + context->secretlen = (UInt32)secret.Length; + //Output + context->outptr = outPtr; + context->outlen = (UInt32)rawHashOutput.Length; + //Hash + Argon2_ErrorCodes result = (Argon2_ErrorCodes)_nativeLibrary.Value.Argon2Hash(&ctx); + //Throw exceptions if error + ThrowOnArgonErr(result); + } + } + + + /// <summary> + /// Compares a raw password, with a salt to a raw hash + /// </summary> + /// <param name="rawPass">Password bytes</param> + /// <param name="salt">Salt bytes</param> + /// <param name="secret">Optional secret that was included in hash</param> + /// <param name="hashBytes">Raw hash bytes</param> + /// <param name="timeCost">Time cost</param> + /// <param name="memCost">Memory cost</param> + /// <param name="parallelism">Degree of parallelism</param> + /// <exception cref="OverflowException"></exception> + /// <exception cref="FormatException"></exception> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="InsufficientMemoryException"></exception> + /// <exception cref="VnArgon2PasswordFormatException"></exception> + /// <returns>True if hashes match</returns> + public static bool Verify2id(ReadOnlySpan<byte> rawPass, ReadOnlySpan<byte> salt, ReadOnlySpan<byte> secret, ReadOnlySpan<byte> hashBytes, + uint timeCost = 2, uint memCost = 65535, uint parallelism = 4) + { + //Alloc data for hash output + using MemoryHandle<byte> outputHandle = PwHeap.Alloc<byte>(hashBytes.Length, true); + //Get pointers + fixed (byte* secretptr = secret, pwd = rawPass, slptr = salt) + { + //Setup context + Argon2_Context ctx; + //Pointer + Argon2_Context* context = &ctx; + context->version = Argon2_version.ARGON2_VERSION_13; + context->m_cost = memCost; + context->t_cost = timeCost; + context->threads = parallelism; + context->lanes = parallelism; + //Default flags + context->flags = ARGON2_DEFAULT_FLAGS; + //Use default memory allocator + context->allocate_cbk = null; + context->free_cbk = null; + //Password + context->pwd = pwd; + context->pwdlen = (uint)rawPass.Length; + //Salt + context->salt = slptr; + context->saltlen = (uint)salt.Length; + //Secret + context->secret = secretptr; + context->secretlen = (uint)secret.Length; + //Output + context->outptr = outputHandle.Base; + context->outlen = (uint)outputHandle.Length; + //Hash + Argon2_ErrorCodes result = (Argon2_ErrorCodes)_nativeLibrary.Value.Argon2Hash(&ctx); + //Throw an excpetion if an error ocurred + ThrowOnArgonErr(result); + } + //Return the comparison + return CryptographicOperations.FixedTimeEquals(outputHandle.Span, hashBytes); + } + + /// <summary> + /// Compares a password to a previously hashed password from this library + /// </summary> + /// <param name="rawPass">Password data</param> + /// <param name="secret">Optional secret that was included in hash</param> + /// <param name="hash">Full hash span</param> + /// <param name="hashLen">Length of hash</param> + /// <exception cref="OverflowException"></exception> + /// <exception cref="FormatException"></exception> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="InsufficientMemoryException"></exception> + /// <exception cref="VnArgon2PasswordFormatException"></exception> + /// <returns>True if the password matches the hash</returns> + public static bool Verify2id(ReadOnlySpan<char> rawPass, ReadOnlySpan<char> hash, ReadOnlySpan<byte> secret) + { + if (!hash.Contains(ID_MODE, StringComparison.Ordinal)) + { + throw new VnArgon2PasswordFormatException("The hash argument supplied is not a valid format and cannot be decoded"); + } + Argon2PasswordEntry entry; + try + { + //Init password breakout struct + entry = new(hash); + } + catch (Exception ex) + { + throw new VnArgon2PasswordFormatException("Password format was not recoverable", ex); + } + //Calculate base64 buffer sizes + int passBase64BufSize = Base64.GetMaxDecodedFromUtf8Length(entry.Hash.Length); + int saltBase64BufSize = Base64.GetMaxDecodedFromUtf8Length(entry.Salt.Length); + int rawPassLen = LocEncoding.GetByteCount(rawPass); + //Alloc buffer for decoded data + using MemoryHandle<byte> rawBufferHandle = Memory.Shared.Alloc<byte>(passBase64BufSize + saltBase64BufSize + rawPassLen, true); + //Split buffers + Span<byte> saltBuf = rawBufferHandle.Span[..saltBase64BufSize]; + Span<byte> passBuf = rawBufferHandle.AsSpan(saltBase64BufSize, passBase64BufSize); + Span<byte> rawPassBuf = rawBufferHandle.AsSpan(saltBase64BufSize + passBase64BufSize, rawPassLen); + { + //Decode salt + if (!Convert.TryFromBase64Chars(entry.Hash, passBuf, out int actualHashLen)) + { + throw new VnArgon2PasswordFormatException("Failed to recover hash bytes"); + } + //Resize pass buff + passBuf = passBuf[..actualHashLen]; + } + //Decode salt + { + if (!Convert.TryFromBase64Chars(entry.Salt, saltBuf, out int actualSaltLen)) + { + throw new VnArgon2PasswordFormatException("Failed to recover salt bytes"); + } + //Resize salt buff + saltBuf = saltBuf[..actualSaltLen]; + } + //encode password bytes + rawPassLen = LocEncoding.GetBytes(rawPass, rawPassBuf); + //Verify password + return Verify2id(rawPassBuf[..rawPassLen], saltBuf, secret, passBuf, entry.TimeCost, entry.MemoryCost, entry.Parallelism); + } + + private static void ThrowOnArgonErr(Argon2_ErrorCodes result) + { + switch (result) + { + //Success + case Argon2_ErrorCodes.ARGON2_OK: + break; + case Argon2_ErrorCodes.ARGON2_OUTPUT_PTR_NULL: + throw new VnArgon2Exception("Pointer to output data was null", result); + case Argon2_ErrorCodes.ARGON2_OUTPUT_TOO_SHORT: + throw new VnArgon2Exception("Output array too short", result); + case Argon2_ErrorCodes.ARGON2_OUTPUT_TOO_LONG: + throw new VnArgon2Exception("Pointer output data too long", result); + case Argon2_ErrorCodes.ARGON2_PWD_TOO_SHORT: + throw new VnArgon2Exception("Password too short", result); + case Argon2_ErrorCodes.ARGON2_PWD_TOO_LONG: + throw new VnArgon2Exception("Password too long", result); + case Argon2_ErrorCodes.ARGON2_SECRET_TOO_SHORT: + case Argon2_ErrorCodes.ARGON2_SALT_TOO_SHORT: + throw new VnArgon2Exception("Salt too short", result); + case Argon2_ErrorCodes.ARGON2_SECRET_TOO_LONG: + case Argon2_ErrorCodes.ARGON2_SALT_TOO_LONG: + throw new VnArgon2Exception("Salt too long", result); + case Argon2_ErrorCodes.ARGON2_TIME_TOO_SMALL: + throw new VnArgon2Exception("Time cost too small", result); + case Argon2_ErrorCodes.ARGON2_TIME_TOO_LARGE: + throw new VnArgon2Exception("Time cost too large", result); + case Argon2_ErrorCodes.ARGON2_MEMORY_TOO_LITTLE: + throw new VnArgon2Exception("Memory cost too small", result); + case Argon2_ErrorCodes.ARGON2_MEMORY_TOO_MUCH: + throw new VnArgon2Exception("Memory cost too large", result); + case Argon2_ErrorCodes.ARGON2_LANES_TOO_FEW: + throw new VnArgon2Exception("Not enough parallelism lanes", result); + case Argon2_ErrorCodes.ARGON2_LANES_TOO_MANY: + throw new VnArgon2Exception("Too many parallelism lanes", result); + case Argon2_ErrorCodes.ARGON2_MEMORY_ALLOCATION_ERROR: + throw new VnArgon2Exception("Memory allocation error", result); + case Argon2_ErrorCodes.ARGON2_PWD_PTR_MISMATCH: + case Argon2_ErrorCodes.ARGON2_SALT_PTR_MISMATCH: + case Argon2_ErrorCodes.ARGON2_SECRET_PTR_MISMATCH: + case Argon2_ErrorCodes.ARGON2_AD_PTR_MISMATCH: + case Argon2_ErrorCodes.ARGON2_FREE_MEMORY_CBK_NULL: + case Argon2_ErrorCodes.ARGON2_ALLOCATE_MEMORY_CBK_NULL: + case Argon2_ErrorCodes.ARGON2_INCORRECT_PARAMETER: + case Argon2_ErrorCodes.ARGON2_INCORRECT_TYPE: + case Argon2_ErrorCodes.ARGON2_OUT_PTR_MISMATCH: + case Argon2_ErrorCodes.ARGON2_THREADS_TOO_FEW: + case Argon2_ErrorCodes.ARGON2_THREADS_TOO_MANY: + case Argon2_ErrorCodes.ARGON2_MISSING_ARGS: + case Argon2_ErrorCodes.ARGON2_ENCODING_FAIL: + case Argon2_ErrorCodes.ARGON2_DECODING_FAIL: + case Argon2_ErrorCodes.ARGON2_THREAD_FAIL: + case Argon2_ErrorCodes.ARGON2_DECODING_LENGTH_FAIL: + case Argon2_ErrorCodes.ARGON2_VERIFY_MISMATCH: + default: + throw new VnArgon2Exception($"Unhandled Argon2 operation {result}", result); + } + } + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/Argon2/VnArgon2Exception.cs b/Hashing.Portable/src/Argon2/VnArgon2Exception.cs new file mode 100644 index 0000000..8a3e2b2 --- /dev/null +++ b/Hashing.Portable/src/Argon2/VnArgon2Exception.cs @@ -0,0 +1,54 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: VnArgon2Exception.cs +* +* VnArgon2Exception.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +/* + * VnArgon2.cs + * Author: Vaughhn Nugent + * Date: July 17, 2021 + * + * Dependencies Argon2. + * https://github.com/P-H-C/phc-winner-argon2 + * + */ + +using System; + +namespace VNLib.Hashing +{ + /// <summary> + /// Argon2 operational exception + /// </summary> + public class VnArgon2Exception : Exception + { + /// <summary> + /// Argon2 error code that caused this exception + /// </summary> + public Argon2_ErrorCodes Errno { get; } + public VnArgon2Exception(string message, Argon2_ErrorCodes errno) : base(message) + { + Errno = errno; + } + public override string Message => $"Argon 2 lib error, code {(int)Errno}, name {Enum.GetName(Errno)}, messsage {base.Message}"; + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/Argon2/VnArgon2PasswordFormatException.cs b/Hashing.Portable/src/Argon2/VnArgon2PasswordFormatException.cs new file mode 100644 index 0000000..37e87a6 --- /dev/null +++ b/Hashing.Portable/src/Argon2/VnArgon2PasswordFormatException.cs @@ -0,0 +1,50 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: VnArgon2PasswordFormatException.cs +* +* VnArgon2PasswordFormatException.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +/* + * VnArgon2.cs + * Author: Vaughhn Nugent + * Date: July 17, 2021 + * + * Dependencies Argon2. + * https://github.com/P-H-C/phc-winner-argon2 + * + */ + +using System; + +namespace VNLib.Hashing +{ + /// <summary> + /// Raised if a verify operation determined the supplied password hash is not in a valid format for this library + /// </summary> + public class VnArgon2PasswordFormatException : Exception + { + public VnArgon2PasswordFormatException(string message) : base(message) { } + + public VnArgon2PasswordFormatException(string message, Exception innerException) : base(message, innerException) + { + } + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/IdentityUtility/HashingExtensions.cs b/Hashing.Portable/src/IdentityUtility/HashingExtensions.cs new file mode 100644 index 0000000..f36b151 --- /dev/null +++ b/Hashing.Portable/src/IdentityUtility/HashingExtensions.cs @@ -0,0 +1,171 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: HashingExtensions.cs +* +* HashingExtensions.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.Memory; + +namespace VNLib.Hashing.IdentityUtility +{ + /// <summary> + /// Contains .NET cryptography hasing library extensions + /// </summary> + public static class HashingExtensions + { + /// <summary> + /// Computes the Base64 hash of the specified data using the + /// specified character encoding, or <see cref="Encoding.UTF8"/> + /// by default. + /// </summary> + /// <param name="hmac"></param> + /// <param name="data">The data to compute the hash of</param> + /// <param name="encoding">The <see cref="Encoding"/> used to encode the character buffer</param> + /// <returns>The base64 UTF8 string of the computed hash of the specified data</returns> + public static string ComputeBase64Hash(this HMAC hmac, ReadOnlySpan<char> data, Encoding encoding = null) + { + encoding ??= Encoding.UTF8; + //Calc hashsize to alloc buffer + int hashBufSize = (hmac.HashSize / 8); + //Calc buffer size + int encBufSize = encoding.GetByteCount(data); + //Alloc buffer for encoding data + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(encBufSize + hashBufSize); + Span<byte> encBuffer = buffer.Span[0..encBufSize]; + Span<byte> hashBuffer = buffer.Span[encBufSize..]; + //Encode data + _ = encoding.GetBytes(data, encBuffer); + //compute hash + if (!hmac.TryComputeHash(encBuffer, hashBuffer, out int hashBytesWritten)) + { + throw new OutOfMemoryException("Hash buffer size was too small"); + } + //Convert to base64 string + return Convert.ToBase64String(hashBuffer[..hashBytesWritten]); + } + /// <summary> + /// Computes the hash of the raw data and compares the computed hash against + /// the specified base64hash + /// </summary> + /// <param name="hmac"></param> + /// <param name="raw">The raw data buffer (encoded characters) to decode and compute the hash of</param> + /// <param name="base64Hmac">The base64 hash to verify against</param> + /// <param name="encoding">The encoding used to encode the raw data balue</param> + /// <returns>A value indicating if the hash values match</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static bool VerifyBase64Hash(this HMAC hmac, ReadOnlySpan<char> base64Hmac, ReadOnlySpan<char> raw, Encoding encoding = null) + { + _ = hmac ?? throw new ArgumentNullException(nameof(hmac)); + if (raw.IsEmpty) + { + throw new ArgumentException("Raw data buffer must not be empty", nameof(raw)); + } + if (base64Hmac.IsEmpty) + { + throw new ArgumentException("Hmac buffer must not be empty", nameof(base64Hmac)); + } + encoding ??= Encoding.UTF8; + //Calc buffer size + int rawDataBufSize = encoding.GetByteCount(raw); + //Calc base64 buffer size + int base64BufSize = base64Hmac.Length; + //Alloc buffer for encoding and raw data + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(rawDataBufSize + base64BufSize, true); + Span<byte> rawDataBuf = buffer.Span[0..rawDataBufSize]; + Span<byte> base64Buf = buffer.Span[rawDataBufSize..]; + //encode + _ = encoding.GetBytes(raw, rawDataBuf); + //Convert to binary + if(!Convert.TryFromBase64Chars(base64Hmac, base64Buf, out int base64Converted)) + { + throw new OutOfMemoryException("Base64 buffer too small"); + } + //Compare hash buffers + return hmac.VerifyHash(base64Buf[0..base64Converted], rawDataBuf); + } + /// <summary> + /// Computes the hash of the raw data and compares the computed hash against + /// the specified hash + /// </summary> + /// <param name="hmac"></param> + /// <param name="raw">The raw data to verify the hash of</param> + /// <param name="hash">The hash to compare against the computed data</param> + /// <returns>A value indicating if the hash values match</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static bool VerifyHash(this HMAC hmac, ReadOnlySpan<byte> hash, ReadOnlySpan<byte> raw) + { + if (raw.IsEmpty) + { + throw new ArgumentException("Raw data buffer must not be empty", nameof(raw)); + } + if (hash.IsEmpty) + { + throw new ArgumentException("Hash buffer must not be empty", nameof(hash)); + } + //Calc hashsize to alloc buffer + int hashBufSize = hmac.HashSize / 8; + //Alloc buffer for hash + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(hashBufSize); + //compute hash + if (!hmac.TryComputeHash(raw, buffer, out int hashBytesWritten)) + { + throw new OutOfMemoryException("Hash buffer size was too small"); + } + //Compare hash buffers + return CryptographicOperations.FixedTimeEquals(buffer.Span[0..hashBytesWritten], hash); + } + + /// <summary> + /// Attempts to encrypt the specified character buffer using the specified encoding + /// </summary> + /// <param name="alg"></param> + /// <param name="data">The data to encrypt</param> + /// <param name="output">The output buffer</param> + /// <param name="padding">The encryption padding to use</param> + /// <param name="enc">Character encoding used to encode the character buffer</param> + /// <returns>The number of bytes encrypted, or 0/false otherwise</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="CryptographicException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static ERRNO TryEncrypt(this RSA alg, ReadOnlySpan<char> data, in Span<byte> output, RSAEncryptionPadding padding, Encoding enc = null) + { + _ = alg ?? throw new ArgumentNullException(nameof(alg)); + //Default to UTF8 encoding + enc ??= Encoding.UTF8; + //Alloc decode buffer + int buffSize = enc.GetByteCount(data); + //Alloc buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(buffSize, true); + //Encode data + int converted = enc.GetBytes(data, buffer); + //Try encrypt + return !alg.TryEncrypt(buffer.Span, output, padding, out int bytesWritten) ? ERRNO.E_FAIL : (ERRNO)bytesWritten; + } + } +} diff --git a/Hashing.Portable/src/IdentityUtility/JsonWebKey.cs b/Hashing.Portable/src/IdentityUtility/JsonWebKey.cs new file mode 100644 index 0000000..54098c2 --- /dev/null +++ b/Hashing.Portable/src/IdentityUtility/JsonWebKey.cs @@ -0,0 +1,460 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: JsonWebKey.cs +* +* JsonWebKey.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Hashing.IdentityUtility +{ + public static class JWKAlgorithms + { + public const string RS256 = "RS256"; + public const string RS384 = "RS384"; + public const string RS512 = "RS512"; + public const string PS256 = "PS256"; + public const string PS384 = "PS384"; + public const string PS512 = "PS512"; + public const string ES256 = "ES256"; + public const string ES384 = "ES384"; + public const string ES512 = "ES512"; + } + + public class EncryptionTypeNotSupportedException : NotSupportedException + { + public EncryptionTypeNotSupportedException(string message) : base(message) + {} + + public EncryptionTypeNotSupportedException(string message, Exception innerException) : base(message, innerException) + {} + + public EncryptionTypeNotSupportedException() + {} + } + + public static class JsonWebKey + { + + /// <summary> + /// Verifies the <see cref="JsonWebToken"/> against the supplied + /// Json Web Key in <see cref="JsonDocument"/> format + /// </summary> + /// <param name="token"></param> + /// <param name="jwk">The supplied single Json Web Key</param> + /// <returns>True if required JWK data exists, ciphers were created, and data is verified, false otherwise</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="EncryptionTypeNotSupportedException"></exception> + public static bool VerifyFromJwk(this JsonWebToken token, in JsonElement jwk) + { + //Get key use and algorithm + string? use = jwk.GetPropString("use"); + string? alg = jwk.GetPropString("alg"); + + //Use and alg are required here + if(use == null || alg == null) + { + return false; + } + + //Make sure the key is used for signing/verification + if (!"sig".Equals(use, StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + using (JsonDocument jwtHeader = token.GetHeader()) + { + string? jwtAlg = jwtHeader.RootElement.GetPropString("alg"); + //Make sure the jwt was signed with the same algorithm type + if (!alg.Equals(jwtAlg, StringComparison.OrdinalIgnoreCase)) + { + return false; + } + } + + switch (alg.ToUpper(null)) + { + //Rsa witj pkcs and pss + case JWKAlgorithms.RS256: + { + using RSA? rsa = GetRSAPublicKey(in jwk); + return rsa != null && token.Verify(rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + } + case JWKAlgorithms.RS384: + { + using RSA? rsa = GetRSAPublicKey(in jwk); + return rsa != null && token.Verify(rsa, HashAlgorithmName.SHA384, RSASignaturePadding.Pkcs1); + } + case JWKAlgorithms.RS512: + { + using RSA? rsa = GetRSAPublicKey(in jwk); + return rsa != null && token.Verify(rsa, HashAlgorithmName.SHA512, RSASignaturePadding.Pkcs1); + } + case JWKAlgorithms.PS256: + { + using RSA? rsa = GetRSAPublicKey(in jwk); + return rsa != null && token.Verify(rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pss); + } + case JWKAlgorithms.PS384: + { + using RSA? rsa = GetRSAPublicKey(in jwk); + return rsa != null && token.Verify(rsa, HashAlgorithmName.SHA384, RSASignaturePadding.Pss); + } + case JWKAlgorithms.PS512: + { + using RSA? rsa = GetRSAPublicKey(in jwk); + return rsa != null && token.Verify(rsa, HashAlgorithmName.SHA512, RSASignaturePadding.Pss); + } + //Eccurves + case JWKAlgorithms.ES256: + { + using ECDsa? eCDsa = GetECDsaPublicKey(in jwk); + return eCDsa != null && token.Verify(eCDsa, HashAlgorithmName.SHA256); + } + case JWKAlgorithms.ES384: + { + using ECDsa? eCDsa = GetECDsaPublicKey(in jwk); + return eCDsa != null && token.Verify(eCDsa, HashAlgorithmName.SHA384); + } + case JWKAlgorithms.ES512: + { + using ECDsa? eCDsa = GetECDsaPublicKey(in jwk); + return eCDsa != null && token.Verify(eCDsa, HashAlgorithmName.SHA512); + } + default: + throw new EncryptionTypeNotSupportedException(); + } + + } + + /// <summary> + /// Verifies the <see cref="JsonWebToken"/> against the supplied + /// <see cref="ReadOnlyJsonWebKey"/> + /// </summary> + /// <param name="token"></param> + /// <param name="jwk">The supplied single Json Web Key</param> + /// <returns>True if required JWK data exists, ciphers were created, and data is verified, false otherwise</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="EncryptionTypeNotSupportedException"></exception> + public static bool VerifyFromJwk(this JsonWebToken token, ReadOnlyJsonWebKey jwk) => token.VerifyFromJwk(jwk.KeyElement); + + /// <summary> + /// Signs the <see cref="JsonWebToken"/> with the supplied JWK json element + /// </summary> + /// <param name="token"></param> + /// <param name="jwk">The JWK in the <see cref="JsonElement"/> </param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="EncryptionTypeNotSupportedException"></exception> + public static void SignFromJwk(this JsonWebToken token, in JsonElement jwk) + { + _ = token ?? throw new ArgumentNullException(nameof(token)); + //Get key use and algorithm + string? use = jwk.GetPropString("use"); + string? alg = jwk.GetPropString("alg"); + + //Use and alg are required here + if (use == null || alg == null) + { + throw new InvalidOperationException("Algorithm or JWK use is null"); + } + + //Make sure the key is used for signing/verification + if (!"sig".Equals(use, StringComparison.OrdinalIgnoreCase)) + { + throw new InvalidOperationException("The JWK cannot be used for signing"); + } + + switch (alg.ToUpper(null)) + { + //Rsa witj pkcs and pss + case JWKAlgorithms.RS256: + { + using RSA? rsa = GetRSAPrivateKey(in jwk); + _ = rsa ?? throw new InvalidOperationException("JWK Does not contain an RSA private key"); + token.Sign(rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1, 128); + return; + } + case JWKAlgorithms.RS384: + { + using RSA? rsa = GetRSAPrivateKey(in jwk); + _ = rsa ?? throw new InvalidOperationException("JWK Does not contain an RSA private key"); + token.Sign(rsa, HashAlgorithmName.SHA384, RSASignaturePadding.Pkcs1, 128); + return; + } + case JWKAlgorithms.RS512: + { + using RSA? rsa = GetRSAPrivateKey(in jwk); + _ = rsa ?? throw new InvalidOperationException("JWK Does not contain an RSA private key"); + token.Sign(rsa, HashAlgorithmName.SHA512, RSASignaturePadding.Pkcs1, 256); + return; + } + case JWKAlgorithms.PS256: + { + using RSA? rsa = GetRSAPrivateKey(in jwk); + _ = rsa ?? throw new InvalidOperationException("JWK Does not contain an RSA private key"); + token.Sign(rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pss, 128); + return; + } + case JWKAlgorithms.PS384: + { + using RSA? rsa = GetRSAPrivateKey(in jwk); + _ = rsa ?? throw new InvalidOperationException("JWK Does not contain an RSA private key"); + token.Sign(rsa, HashAlgorithmName.SHA384, RSASignaturePadding.Pss, 128); + return; + } + case JWKAlgorithms.PS512: + { + using RSA? rsa = GetRSAPrivateKey(in jwk); + _ = rsa ?? throw new InvalidOperationException("JWK Does not contain an RSA private key"); + token.Sign(rsa, HashAlgorithmName.SHA512, RSASignaturePadding.Pss, 256); + return; + } + //Eccurves + case JWKAlgorithms.ES256: + { + using ECDsa? eCDsa = GetECDsaPrivateKey(in jwk); + _ = eCDsa ?? throw new InvalidOperationException("JWK Does not contain an ECDsa private key"); + token.Sign(eCDsa, HashAlgorithmName.SHA256, 128); + return; + } + case JWKAlgorithms.ES384: + { + using ECDsa? eCDsa = GetECDsaPrivateKey(in jwk); + _ = eCDsa ?? throw new InvalidOperationException("JWK Does not contain an ECDsa private key"); + token.Sign(eCDsa, HashAlgorithmName.SHA384, 128); + return; + } + case JWKAlgorithms.ES512: + { + using ECDsa? eCDsa = GetECDsaPrivateKey(in jwk); + _ = eCDsa ?? throw new InvalidOperationException("JWK Does not contain an ECDsa private key"); + token.Sign(eCDsa, HashAlgorithmName.SHA512, 256); + return; + } + default: + throw new EncryptionTypeNotSupportedException(); + } + } + + /// <summary> + /// Signs the <see cref="JsonWebToken"/> with the supplied JWK json element + /// </summary> + /// <param name="token"></param> + /// <param name="jwk">The JWK in the <see cref="ReadOnlyJsonWebKey"/> </param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="EncryptionTypeNotSupportedException"></exception> + public static void SignFromJwk(this JsonWebToken token, ReadOnlyJsonWebKey jwk) => token.SignFromJwk(jwk.KeyElement); + + /// <summary> + /// Gets the <see cref="RSA"/> public key algorithm for the current <see cref="ReadOnlyJsonWebKey"/> + /// </summary> + /// <param name="key"></param> + /// <returns>The <see cref="RSA"/> algorithm of the public key if loaded</returns> + public static RSA? GetRSAPublicKey(this ReadOnlyJsonWebKey key) => key == null ? null : GetRSAPublicKey(key.KeyElement); + + /// <summary> + /// Gets the <see cref="RSA"/> private key algorithm for the current <see cref="ReadOnlyJsonWebKey"/> + /// </summary> + /// <param name="key"></param> + ///<returns>The <see cref="RSA"/> algorithm of the private key key if loaded</returns> + public static RSA? GetRSAPrivateKey(this ReadOnlyJsonWebKey key) => key == null ? null : GetRSAPrivateKey(key.KeyElement); + + /// <summary> + /// Gets the RSA public key algorithm from the supplied Json Web Key <see cref="JsonElement"/> + /// </summary> + /// <param name="jwk">The element that contains the JWK data</param> + /// <returns>The <see cref="RSA"/> algorithm if found, or null if the element does not contain public key</returns> + public static RSA? GetRSAPublicKey(in JsonElement jwk) + { + RSAParameters? rSAParameters = GetRsaParameters(in jwk, false); + //Create rsa from params + return rSAParameters.HasValue ? RSA.Create(rSAParameters.Value) : null; + } + + /// <summary> + /// Gets the RSA private key algorithm from the supplied Json Web Key <see cref="JsonElement"/> + /// </summary> + /// <param name="jwk"></param> + /// <returns>The <see cref="RSA"/> algorithm if found, or null if the element does not contain private key</returns> + public static RSA? GetRSAPrivateKey(in JsonElement jwk) + { + RSAParameters? rSAParameters = GetRsaParameters(in jwk, true); + //Create rsa from params + return rSAParameters.HasValue ? RSA.Create(rSAParameters.Value) : null; + } + + private static RSAParameters? GetRsaParameters(in JsonElement jwk, bool includePrivateKey) + { + //Get the RSA public key credentials + ReadOnlySpan<char> e = jwk.GetPropString("e"); + ReadOnlySpan<char> n = jwk.GetPropString("n"); + + if (e.IsEmpty || n.IsEmpty) + { + return null; + } + + if (includePrivateKey) + { + //Get optional private key params + ReadOnlySpan<char> d = jwk.GetPropString("d"); + ReadOnlySpan<char> dp = jwk.GetPropString("dq"); + ReadOnlySpan<char> dq = jwk.GetPropString("dp"); + ReadOnlySpan<char> p = jwk.GetPropString("p"); + ReadOnlySpan<char> q = jwk.GetPropString("q"); + + //Create params from exponent, moduls and private key components + return new() + { + Exponent = FromBase64UrlChars(e), + Modulus = FromBase64UrlChars(n), + D = FromBase64UrlChars(d), + DP = FromBase64UrlChars(dp), + DQ = FromBase64UrlChars(dq), + P = FromBase64UrlChars(p), + Q = FromBase64UrlChars(q), + }; + } + else + { + //Create params from exponent and moduls + return new() + { + Exponent = FromBase64UrlChars(e), + Modulus = FromBase64UrlChars(n), + }; + } + + } + + + /// <summary> + /// Gets the ECDsa public key algorithm for the current <see cref="ReadOnlyJsonWebKey"/> + /// </summary> + /// <param name="key"></param> + /// <returns>The <see cref="ECDsa"/> algorithm of the public key if loaded</returns> + public static ECDsa? GetECDsaPublicKey(this ReadOnlyJsonWebKey key) => key == null ? null : GetECDsaPublicKey(key.KeyElement); + + /// <summary> + /// Gets the <see cref="ECDsa"/> private key algorithm for the current <see cref="ReadOnlyJsonWebKey"/> + /// </summary> + /// <param name="key"></param> + ///<returns>The <see cref="ECDsa"/> algorithm of the private key key if loaded</returns> + public static ECDsa? GetECDsaPrivateKey(this ReadOnlyJsonWebKey key) => key == null ? null : GetECDsaPrivateKey(key.KeyElement); + + /// <summary> + /// Gets the ECDsa public key algorithm from the supplied Json Web Key <see cref="JsonElement"/> + /// </summary> + /// <param name="jwk">The public key element</param> + /// <returns>The <see cref="ECDsa"/> algorithm from the key if loaded, null if no key data was found</returns> + public static ECDsa? GetECDsaPublicKey(in JsonElement jwk) + { + //Get the EC params + ECParameters? ecParams = GetECParameters(in jwk, false); + //Return new alg + return ecParams.HasValue ? ECDsa.Create(ecParams.Value) : null; + } + + /// <summary> + /// Gets the ECDsa private key algorithm from the supplied Json Web Key <see cref="JsonElement"/> + /// </summary> + /// <param name="jwk">The element that contains the private key data</param> + /// <returns>The <see cref="ECDsa"/> algorithm from the key if loaded, null if no key data was found</returns> + public static ECDsa? GetECDsaPrivateKey(in JsonElement jwk) + { + //Get the EC params + ECParameters? ecParams = GetECParameters(in jwk, true); + //Return new alg + return ecParams.HasValue ? ECDsa.Create(ecParams.Value) : null; + } + + + private static ECParameters? GetECParameters(in JsonElement jwk, bool includePrivate) + { + //Get the RSA public key credentials + ReadOnlySpan<char> x = jwk.GetPropString("x"); + ReadOnlySpan<char> y = jwk.GetPropString("y"); + + //Optional private key + ReadOnlySpan<char> d = includePrivate ? jwk.GetPropString("d") : null; + + if (x.IsEmpty || y.IsEmpty) + { + return null; + } + + ECCurve curve; + //Get the EC curve name from the curve ID + switch (jwk.GetPropString("crv")?.ToUpper(null)) + { + case "P-256": + curve = ECCurve.NamedCurves.nistP256; + break; + case "P-384": + curve = ECCurve.NamedCurves.nistP384; + break; + case "P-521": + curve = ECCurve.NamedCurves.nistP521; + break; + default: + return null; + } + + //get params + return new() + { + Curve = curve, + Q = new ECPoint() + { + X = FromBase64UrlChars(x), + Y = FromBase64UrlChars(y), + }, + //Optional private key + D = FromBase64UrlChars(d) + }; + } + + private static byte[]? FromBase64UrlChars(ReadOnlySpan<char> base64) + { + if (base64.IsEmpty) + { + return null; + } + //bin buffer for temp decoding + using UnsafeMemoryHandle<byte> binBuffer = Memory.UnsafeAlloc<byte>(base64.Length + 16, false); + //base64url decode + ERRNO count = VnEncoding.Base64UrlDecode(base64, binBuffer.Span); + //Return buffer or null if failed + return count ? binBuffer.AsSpan(0, count).ToArray() : null; + } + } +} diff --git a/Hashing.Portable/src/IdentityUtility/JsonWebToken.cs b/Hashing.Portable/src/IdentityUtility/JsonWebToken.cs new file mode 100644 index 0000000..716dd4c --- /dev/null +++ b/Hashing.Portable/src/IdentityUtility/JsonWebToken.cs @@ -0,0 +1,385 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: JsonWebToken.cs +* +* JsonWebToken.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Buffers; +using System.Buffers.Text; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Hashing.IdentityUtility +{ + /// <summary> + /// Provides a dynamic JSON Web Token class that will store and + /// compute Base64Url encoded WebTokens + /// </summary> + public class JsonWebToken : VnDisposeable, IStringSerializeable, IDisposable + { + internal const byte SAEF_PERIOD = 0x2e; + internal const byte PADDING_BYTES = 0x3d; + + /// <summary> + /// Parses a JWT from a Base64URL encoded character buffer + /// </summary> + /// <param name="urlEncJwtString"></param> + /// <param name="heap">An optional <see cref="IUnmangedHeap"/> instance to alloc buffers from</param> + /// <returns>The parses <see cref="JsonWebToken"/></returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static JsonWebToken Parse(ReadOnlySpan<char> urlEncJwtString, IUnmangedHeap heap = null) + { + heap ??= Memory.Shared; + //Calculate the decoded size of the characters to alloc a buffer + int utf8Size = Encoding.UTF8.GetByteCount(urlEncJwtString); + //Alloc bin buffer to store decode data + using MemoryHandle<byte> binBuffer = heap.Alloc<byte>(utf8Size, true); + //Decode to utf8 + utf8Size = Encoding.UTF8.GetBytes(urlEncJwtString, binBuffer); + //Parse and return the jwt + return ParseRaw(binBuffer.Span[..utf8Size], heap); + } + + /// <summary> + /// Parses a buffer of UTF8 bytes of url encoded base64 characters + /// </summary> + /// <param name="utf8JWTData">The JWT data buffer</param> + /// <param name="heap">An optional <see cref="IUnmangedHeap"/> instance to alloc buffers from</param> + /// <returns>The parsed <see cref="JsonWebToken"/></returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static JsonWebToken ParseRaw(ReadOnlySpan<byte> utf8JWTData, IUnmangedHeap heap = null) + { + if (utf8JWTData.IsEmpty) + { + throw new ArgumentException("JWT data may not be empty", nameof(utf8JWTData)); + } + //Set default heap of non was specified + heap ??= Memory.Shared; + //Alloc the token and copy the supplied data to a new mem stream + JsonWebToken jwt = new(heap, new (heap, utf8JWTData)); + try + { + ReadOnlySpan<byte> buffer = jwt.DataBuffer; + //Search for the first period to indicate the end of the header section + jwt.HeaderEnd = buffer.IndexOf(SAEF_PERIOD); + //Make sure a '.' was found + if (jwt.HeaderEnd < 0) + { + throw new FormatException("The supplied data is not a valid Json Web Token, header end symbol could not be found"); + } + //Shift buffer window + buffer = buffer[jwt.PayloadStart..]; + //Search for next period to end the payload + jwt.PayloadEnd = jwt.PayloadStart + buffer.LastIndexOf(SAEF_PERIOD); + //Make sure a '.' was found + if (jwt.PayloadEnd < 0) + { + throw new FormatException("The supplied data is not a valid Json Web Token, payload end symbol could not be found"); + } + //signature is set automatically + //return the new token + return jwt; + } + catch + { + jwt.Dispose(); + throw; + } + } + + + /// <summary> + /// The heap used to allocate buffers from + /// </summary> + public IUnmangedHeap Heap { get; } + /// <summary> + /// The size (in bytes) of the encoded data that makes + /// up the current JWT. + /// </summary> + public int ByteSize => (int)DataStream.Position; + /// <summary> + /// A buffer that represents the current state of the JWT buffer. + /// Utf8Base64Url encoded data. + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + public ReadOnlySpan<byte> DataBuffer => DataStream.AsSpan()[..ByteSize]; + + + private readonly VnMemoryStream DataStream; + + /// <summary> + /// Creates a new <see cref="JsonWebToken"/> with the specified initial state + /// </summary> + /// <param name="heap">The heap used to alloc buffers</param> + /// <param name="initialData">The initial data of the jwt</param> + protected JsonWebToken(IUnmangedHeap heap, VnMemoryStream initialData) + { + Heap = heap; + DataStream = initialData; + initialData.Position = initialData.Length; + } + + /// <summary> + /// Creates a new empty JWT instance, with an optional heap to alloc + /// buffers from. (<see cref="Memory.Shared"/> is used as default) + /// </summary> + /// <param name="heap">The <see cref="IUnmangedHeap"/> to alloc buffers from</param> + public JsonWebToken(IUnmangedHeap heap = null) + { + Heap = heap ?? Memory.Shared; + DataStream = new(Heap, 100, true); + } + + #region Header + private int HeaderEnd; + /// <summary> + /// The Base64URL encoded UTF8 bytes of the header portion of the current JWT + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public ReadOnlySpan<byte> HeaderData => DataBuffer[..HeaderEnd]; + + /// <summary> + /// Encodes and stores the specified header value to the begining of the + /// JWT. This method may only be called once, if the header has not already been supplied. + /// </summary> + /// <param name="header">The value of the JWT header parameter</param> + /// <exception cref="OutOfMemoryException"></exception> + public void WriteHeader(ReadOnlySpan<byte> header) + { + //reset the buffer + DataStream.Position = 0; + //Write the header data + WriteValue(header); + //The header end is the position of the stream since it was empty + HeaderEnd = ByteSize; + } + + #endregion + + #region Payload + private int PayloadStart => HeaderEnd + 1; + private int PayloadEnd; + /// <summary> + /// The Base64URL encoded UTF8 bytes of the payload portion of the current JWT + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public ReadOnlySpan<byte> PayloadData => DataBuffer[PayloadStart..PayloadEnd]; + + /// <summary> + /// The Base64URL encoded UTF8 bytes of the header + '.' + payload portion of the current jwt + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public ReadOnlySpan<byte> HeaderAndPayload => DataBuffer[..PayloadEnd]; + + /// <summary> + /// Encodes and stores the specified payload data and appends it to the current + /// JWT buffer. This method may only be called once, if the header has not already been supplied. + /// </summary> + /// <param name="payload">The value of the JWT payload section</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public void WritePayload(ReadOnlySpan<byte> payload) + { + //Write leading period + DataStream.WriteByte(SAEF_PERIOD); + //Write payload + WriteValue(payload); + //Store final position + PayloadEnd = ByteSize; + } + /// <summary> + /// Encodes the specified value and writes it to the + /// internal buffer + /// </summary> + /// <param name="value">The data value to encode and buffer</param> + /// <exception cref="OutOfMemoryException"></exception> + protected void WriteValue(ReadOnlySpan<byte> value) + { + //Calculate the proper base64 buffer size + int base64BufSize = Base64.GetMaxEncodedToUtf8Length(value.Length); + //Alloc buffer + using UnsafeMemoryHandle<byte> binBuffer = Heap.UnsafeAlloc<byte>(base64BufSize); + //Slice off the begiing of the buffer for the base64 encoding + if(Base64.EncodeToUtf8(value, binBuffer.Span, out _, out int written) != OperationStatus.Done) + { + throw new OutOfMemoryException(); + } + //Base64 encoded + Span<byte> base64Data = binBuffer.Span[..written].Trim(PADDING_BYTES); + //Convert to rfc4648 urlsafe version + VnEncoding.Base64ToUrlSafeInPlace(base64Data); + //Write the endoded buffer to the stream + DataStream.Write(base64Data); + } + #endregion + + #region Signature + private int SignatureStart => PayloadEnd + 1; + private int SignatureEnd => ByteSize; + /// <summary> + /// The Base64URL encoded UTF8 bytes of the signature portion of the current JWT + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public ReadOnlySpan<byte> SignatureData => DataBuffer[SignatureStart..SignatureEnd]; + + /// <summary> + /// Signs the current JWT (header + payload) data + /// and writes the signature the end of the current buffer, + /// using the specified <see cref="HashAlgorithm"/>. + /// </summary> + /// <param name="signatureAlgorithm">An alternate <see cref="HashAlgorithm"/> instance to sign the JWT with</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public virtual void Sign(HashAlgorithm signatureAlgorithm) + { + Check(); + _ = signatureAlgorithm ?? throw new ArgumentNullException(nameof(signatureAlgorithm)); + //Calculate the size of the buffer to use for the current algorithm + int bufferSize = signatureAlgorithm.HashSize / 8; + //Alloc buffer for signature output + Span<byte> signatureBuffer = stackalloc byte[bufferSize]; + //Compute the hash of the current payload + if(!signatureAlgorithm.TryComputeHash(DataBuffer, signatureBuffer, out int bytesWritten)) + { + throw new OutOfMemoryException(); + } + //Reset the stream position to the end of the payload + DataStream.SetLength(PayloadEnd); + //Write leading period + DataStream.WriteByte(SAEF_PERIOD); + //Write the signature data to the buffer + WriteValue(signatureBuffer[..bytesWritten]); + } + /// <summary> + /// Use an RSA algorithm to sign the JWT message + /// </summary> + /// <param name="rsa">The algorithm used to sign the token</param> + /// <param name="hashAlg">The hash algorithm to use</param> + /// <param name="padding">The signature padding to use</param> + /// <param name="hashSize">The size (in bytes) of the hash output</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public virtual void Sign(RSA rsa, in HashAlgorithmName hashAlg, RSASignaturePadding padding, int hashSize) + { + Check(); + _ = rsa ?? throw new ArgumentNullException(nameof(rsa)); + //Calculate the size of the buffer to use for the current algorithm + using UnsafeMemoryHandle<byte> sigBuffer = Heap.UnsafeAlloc<byte>(hashSize); + if(!rsa.TrySignData(HeaderAndPayload, sigBuffer.Span, hashAlg, padding, out int hashBytesWritten)) + { + throw new OutOfMemoryException("Signature buffer is not large enough to store the hash"); + } + //Reset the stream position to the end of the payload + DataStream.SetLength(PayloadEnd); + //Write leading period + DataStream.WriteByte(SAEF_PERIOD); + //Write the signature data to the buffer + WriteValue(sigBuffer.Span[..hashBytesWritten]); + } + /// <summary> + /// Use an RSA algorithm to sign the JWT message + /// </summary> + /// <param name="alg">The algorithm used to sign the token</param> + /// <param name="hashAlg">The hash algorithm to use</param> + /// <param name="hashSize">The size (in bytes) of the hash output</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public virtual void Sign(ECDsa alg, in HashAlgorithmName hashAlg, int hashSize) + { + Check(); + _ = alg ?? throw new ArgumentNullException(nameof(alg)); + //Calculate the size of the buffer to use for the current algorithm + using UnsafeMemoryHandle<byte> sigBuffer = Heap.UnsafeAlloc<byte>(hashSize); + if (!alg.TrySignData(HeaderAndPayload, sigBuffer.Span, hashAlg, out int hashBytesWritten)) + { + throw new OutOfMemoryException("Signature buffer is not large enough to store the hash"); + } + //Reset the stream position to the end of the payload + DataStream.SetLength(PayloadEnd); + //Write leading period + DataStream.WriteByte(SAEF_PERIOD); + //Write the signature data to the buffer + WriteValue(sigBuffer.Span[..hashBytesWritten]); + } + + #endregion + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public virtual string Compile() => Encoding.UTF8.GetString(DataBuffer); + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public virtual void Compile(ref ForwardOnlyWriter<char> writer) => _ = Encoding.UTF8.GetChars(DataBuffer, ref writer); + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public virtual ERRNO Compile(in Span<char> buffer) + { + ForwardOnlyWriter<char> writer = new(buffer); + Compile(ref writer); + return writer.Written; + } + + /// <summary> + /// Reset's the internal JWT buffer + /// </summary> + public virtual void Reset() + { + DataStream.Position = 0; + //Reset segment indexes + HeaderEnd = 0; + PayloadEnd = 0; + } + + /// <summary> + /// Compiles the current JWT instance and converts it to a string + /// </summary> + /// <returns>A Base64Url enocded string of the JWT format</returns> + public override string ToString() => Compile(); + + ///<inheritdoc/> + protected override void Free() + { + //Clear pointers, so buffer get operations just return empty instead of throwing + Reset(); + DataStream.Dispose(); + } + + } +} diff --git a/Hashing.Portable/src/IdentityUtility/JwtClaim.cs b/Hashing.Portable/src/IdentityUtility/JwtClaim.cs new file mode 100644 index 0000000..55610a5 --- /dev/null +++ b/Hashing.Portable/src/IdentityUtility/JwtClaim.cs @@ -0,0 +1,72 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: JwtClaim.cs +* +* JwtClaim.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System.Collections.Generic; + +using VNLib.Utils; + +namespace VNLib.Hashing.IdentityUtility +{ + /// <summary> + /// A fluent api structure for adding and committing claims to a <see cref="JsonWebToken"/> + /// </summary> + public readonly struct JwtPayload : IIndexable<string, object> + { + private readonly Dictionary<string, object> Claims; + private readonly JsonWebToken Jwt; + + internal JwtPayload(JsonWebToken jwt, int initialCapacity) + { + Jwt = jwt; + Claims = new(initialCapacity); + } + + ///<inheritdoc/> + public object this[string key] + { + get => Claims[key]; + set => Claims[key] = value; + } + + /// <summary> + /// Adds a claim name-value pair to the store + /// </summary> + /// <param name="claim">The clame name</param> + /// <param name="value">The value of the claim</param> + /// <returns>The chained response object</returns> + public JwtPayload AddClaim(string claim, object value) + { + Claims.Add(claim, value); + return this; + } + /// <summary> + /// Writes all claims to the <see cref="JsonWebToken"/> payload segment + /// </summary> + public void CommitClaims() + { + Jwt.WritePayload(Claims); + Claims.Clear(); + } + } +} diff --git a/Hashing.Portable/src/IdentityUtility/JwtExtensions.cs b/Hashing.Portable/src/IdentityUtility/JwtExtensions.cs new file mode 100644 index 0000000..fca2e75 --- /dev/null +++ b/Hashing.Portable/src/IdentityUtility/JwtExtensions.cs @@ -0,0 +1,269 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: JwtExtensions.cs +* +* JwtExtensions.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Text.Json; +using System.Buffers.Text; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Hashing.IdentityUtility +{ + /// <summary> + /// Provides extension methods for manipulating + /// and verifying <see cref="JsonWebToken"/>s + /// </summary> + public static class JwtExtensions + { + /// <summary> + /// Writes the message header as the specified object + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="jwt"></param> + /// <param name="header">The header object</param> + /// <param name="jso">Optional serialize options</param> + public static void WriteHeader<T>(this JsonWebToken jwt, T header, JsonSerializerOptions? jso = null) where T: class + { + byte[] data = JsonSerializer.SerializeToUtf8Bytes(header, jso); + jwt.WriteHeader(data); + } + /// <summary> + /// Writes the message payload as the specified object + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="jwt"></param> + /// <param name="payload">The payload object</param> + /// <param name="jso">Optional serialize options</param> + public static void WritePayload<T>(this JsonWebToken jwt, T payload, JsonSerializerOptions? jso = null) + { + byte[] data = JsonSerializer.SerializeToUtf8Bytes(payload, jso); + jwt.WritePayload(data); + } + + /// <summary> + /// Gets the payload data as a <see cref="JsonDocument"/> + /// </summary> + /// <param name="jwt"></param> + /// <returns>The <see cref="JsonDocument"/> of the jwt body</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="FormatException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static JsonDocument GetPayload(this JsonWebToken jwt) + { + ReadOnlySpan<byte> payload = jwt.PayloadData; + if (payload.IsEmpty) + { + return JsonDocument.Parse("{}"); + } + //calc padding bytes to add + int paddingToAdd = CalcPadding(payload.Length); + //Alloc buffer to copy jwt payload data to + using UnsafeMemoryHandle<byte> buffer = jwt.Heap.UnsafeAlloc<byte>(payload.Length + paddingToAdd); + //Decode from urlsafe base64 + int decoded = DecodeUnpadded(payload, buffer.Span); + //Get json reader to read the first token (payload object) and return a document around it + Utf8JsonReader reader = new(buffer.Span[..decoded]); + return JsonDocument.ParseValue(ref reader); + } + + /// <summary> + /// Gets the header data as a <see cref="JsonDocument"/> + /// </summary> + /// <param name="jwt"></param> + /// <returns>The <see cref="JsonDocument"/> of the jwt body</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="FormatException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static JsonDocument GetHeader(this JsonWebToken jwt) + { + ReadOnlySpan<byte> header = jwt.HeaderData; + if (header.IsEmpty) + { + return JsonDocument.Parse("{}"); + } + //calc padding bytes to add + int paddingToAdd = CalcPadding(header.Length); + //Alloc buffer to copy jwt header data to + using UnsafeMemoryHandle<byte> buffer = jwt.Heap.UnsafeAlloc<byte>(header.Length + paddingToAdd); + //Decode from urlsafe base64 + int decoded = DecodeUnpadded(header, buffer.Span); + //Get json reader to read the first token (payload object) and return a document around it + Utf8JsonReader reader = new(buffer.Span[..decoded]); + return JsonDocument.ParseValue(ref reader); + } + + /* + * Determines how many padding bytes to add at the end + * of the base64 unpadded buffer + * + * Decodes the base64url to base64, then back to its binary + */ + private static int CalcPadding(int length) => (4 - (length % 4)) & 0x03; + private static int DecodeUnpadded(ReadOnlySpan<byte> prePadding, Span<byte> output) + { + ERRNO count = VnEncoding.Base64UrlDecode(prePadding, output); + return count ? count : throw new FormatException($"Failed to decode the utf8 encoded data"); + } + + /// <summary> + /// Deserialzes the jwt payload as the specified object + /// </summary> + /// <param name="jwt"></param> + /// <param name="jso">Optional serialzie options</param> + /// <returns>The <see cref="JsonDocument"/> of the jwt body</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static T? GetPayload<T>(this JsonWebToken jwt, JsonSerializerOptions? jso = null) + { + ReadOnlySpan<byte> payload = jwt.PayloadData; + if (payload.IsEmpty) + { + return default; + } + //calc padding bytes to add + int paddingToAdd = CalcPadding(payload.Length); + //Alloc buffer to copy jwt payload data to + using UnsafeMemoryHandle<byte> buffer = jwt.Heap.UnsafeAlloc<byte>(payload.Length + paddingToAdd); + //Decode from urlsafe base64 + int decoded = DecodeUnpadded(payload, buffer.Span); + //Deserialze as an object + return JsonSerializer.Deserialize<T>(buffer.Span[..decoded], jso); + } + + /// <summary> + /// Verifies the current JWT body-segements against the parsed signature segment. + /// </summary> + /// <param name="jwt"></param> + /// <param name="verificationAlg"> + /// The <see cref="HashAlgorithm"/> to use when calculating the hash of the JWT + /// </param> + /// <returns> + /// True if the signature field of the current JWT matches the re-computed signature of the header and data-fields + /// signature + /// </returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static bool Verify(this JsonWebToken jwt, HashAlgorithm verificationAlg) + { + _ = jwt ?? throw new ArgumentNullException(nameof(jwt)); + _ = verificationAlg ?? throw new ArgumentNullException(nameof(verificationAlg)); + + //Calculate the size of the buffer to use for the current algorithm and make sure it will include the utf8 encoding + int hashBufferSize = Base64.GetMaxEncodedToUtf8Length(verificationAlg.HashSize / 8); + + //Alloc buffer for signature output + Span<byte> signatureBuffer = stackalloc byte[hashBufferSize]; + + //Compute the hash of the current payload + if (!verificationAlg.TryComputeHash(jwt.HeaderAndPayload, signatureBuffer, out int bytesWritten)) + { + throw new InternalBufferTooSmallException("Failed to compute the hash of the JWT data"); + } + + //Do an in-place base64 conversion of the signature to base64 + if (Base64.EncodeToUtf8InPlace(signatureBuffer, bytesWritten, out int base64BytesWritten) != OperationStatus.Done) + { + throw new InternalBufferTooSmallException("Failed to convert the signature buffer to its base64 because the buffer was too small"); + } + + //Trim padding + Span<byte> base64 = signatureBuffer[..base64BytesWritten].Trim(JsonWebToken.PADDING_BYTES); + + //Urlencode + VnEncoding.Base64ToUrlSafeInPlace(base64); + + //Verify the signatures and return results + return CryptographicOperations.FixedTimeEquals(jwt.SignatureData, base64); + } + /// <summary> + /// Verifies the signature of the data using the specified <see cref="RSA"/> and hash parameters + /// </summary> + /// <param name="jwt"></param> + /// <param name="alg">The RSA algorithim to use while verifying the signature of the payload</param> + /// <param name="hashAlg">The <see cref="HashAlgorithmName"/> used to hash the signature</param> + /// <param name="padding">The RSA signature padding method</param> + /// <returns>True if the singature has been verified, false otherwise</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static bool Verify(this JsonWebToken jwt, RSA alg, in HashAlgorithmName hashAlg, RSASignaturePadding padding) + { + _ = jwt ?? throw new ArgumentNullException(nameof(jwt)); + _ = alg ?? throw new ArgumentNullException(nameof(alg)); + //Decode the signature + ReadOnlySpan<byte> signature = jwt.SignatureData; + int paddBytes = CalcPadding(signature.Length); + //Alloc buffer to decode data + using UnsafeMemoryHandle<byte> buffer = jwt.Heap.UnsafeAlloc<byte>(signature.Length + paddBytes); + //Decode from urlsafe base64 + int decoded = DecodeUnpadded(signature, buffer.Span); + //Verify signature + return alg.VerifyData(jwt.HeaderAndPayload, buffer.Span[..decoded], hashAlg, padding); + } + /// <summary> + /// Verifies the signature of the data using the specified <see cref="RSA"/> and hash parameters + /// </summary> + /// <param name="jwt"></param> + /// <param name="alg">The RSA algorithim to use while verifying the signature of the payload</param> + /// <param name="hashAlg">The <see cref="HashAlgorithmName"/> used to hash the signature</param> + /// <returns>True if the singature has been verified, false otherwise</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static bool Verify(this JsonWebToken jwt, ECDsa alg, in HashAlgorithmName hashAlg) + { + _ = alg ?? throw new ArgumentNullException(nameof(alg)); + //Decode the signature + ReadOnlySpan<byte> signature = jwt.SignatureData; + int paddBytes = CalcPadding(signature.Length); + //Alloc buffer to decode data + using UnsafeMemoryHandle<byte> buffer = jwt.Heap.UnsafeAlloc<byte>(signature.Length + paddBytes); + //Decode from urlsafe base64 + int decoded = DecodeUnpadded(signature, buffer.Span); + //Verify signature + return alg.VerifyData(jwt.HeaderAndPayload, buffer.Span[..decoded], hashAlg); + } + + /// <summary> + /// Initializes a new <see cref="JwtPayload"/> object for writing claims to the + /// current tokens payload segment + /// </summary> + /// <param name="jwt"></param> + /// <param name="initCapacity">The inital cliam capacity</param> + /// <returns>The fluent chainable stucture</returns> + public static JwtPayload InitPayloadClaim(this JsonWebToken jwt, int initCapacity = 0) => new (jwt, initCapacity); + } +} diff --git a/Hashing.Portable/src/IdentityUtility/ReadOnlyJsonWebKey.cs b/Hashing.Portable/src/IdentityUtility/ReadOnlyJsonWebKey.cs new file mode 100644 index 0000000..f86855a --- /dev/null +++ b/Hashing.Portable/src/IdentityUtility/ReadOnlyJsonWebKey.cs @@ -0,0 +1,128 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: ReadOnlyJsonWebKey.cs +* +* ReadOnlyJsonWebKey.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; + +using VNLib.Utils; +using VNLib.Utils.Extensions; +using System.Collections.Generic; + +namespace VNLib.Hashing.IdentityUtility +{ + /// <summary> + /// A readonly Json Web Key (JWK) data structure that may be used for signing + /// or verifying messages. + /// </summary> + public sealed class ReadOnlyJsonWebKey : VnDisposeable + { + private readonly JsonElement _jwk; + private readonly JsonDocument? doc; + + /// <summary> + /// Creates a new instance of <see cref="ReadOnlyJsonWebKey"/> from a <see cref="JsonElement"/>. + /// This will call <see cref="JsonElement.Clone"/> on the element and store an internal copy + /// </summary> + /// <param name="keyElement">The <see cref="JsonElement"/> to create the <see cref="ReadOnlyJsonWebKey"/> from</param> + public ReadOnlyJsonWebKey(in JsonElement keyElement) + { + _jwk = keyElement.Clone(); + //Set initial values + KeyId = _jwk.GetPropString("kid"); + KeyType = _jwk.GetPropString("kty"); + Algorithm = _jwk.GetPropString("alg"); + Use = _jwk.GetPropString("use"); + + //Create a JWT header from the values + JwtHeader = new Dictionary<string, string?>() + { + { "alg" , Algorithm }, + { "typ" , "JWT" }, + }; + } + + /// <summary> + /// Creates a new instance of <see cref="ReadOnlyJsonWebKey"/> from a raw utf8 encoded json + /// binary sequence + /// </summary> + /// <param name="rawValue">The utf8 encoded json binary sequence</param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="JsonException"></exception> + public ReadOnlyJsonWebKey(ReadOnlySpan<byte> rawValue) + { + //Pare the raw value + Utf8JsonReader reader = new (rawValue); + doc = JsonDocument.ParseValue(ref reader); + //store element + _jwk = doc.RootElement; + + //Set initial values + KeyId = _jwk.GetPropString("kid"); + KeyType = _jwk.GetPropString("kty"); + Algorithm = _jwk.GetPropString("alg"); + Use = _jwk.GetPropString("use"); + + //Create a JWT header from the values + JwtHeader = new Dictionary<string, string?>() + { + { "alg" , Algorithm }, + { "typ" , "JWT" }, + }; + } + + /// <summary> + /// The key identifier + /// </summary> + public string? KeyId { get; } + /// <summary> + /// The key type + /// </summary> + public string? KeyType { get; } + /// <summary> + /// The key algorithm + /// </summary> + public string? Algorithm { get; } + /// <summary> + /// The key "use" value + /// </summary> + public string? Use { get; } + + /// <summary> + /// Returns the JWT header that matches this key + /// </summary> + public IReadOnlyDictionary<string, string?> JwtHeader { get; } + + /// <summary> + /// The key element + /// </summary> + internal JsonElement KeyElement => _jwk; + + ///<inheritdoc/> + protected override void Free() + { + doc?.Dispose(); + } + + } +} diff --git a/Hashing.Portable/src/ManagedHash.cs b/Hashing.Portable/src/ManagedHash.cs new file mode 100644 index 0000000..6cb4426 --- /dev/null +++ b/Hashing.Portable/src/ManagedHash.cs @@ -0,0 +1,365 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: ManagedHash.cs +* +* ManagedHash.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.Memory; + +namespace VNLib.Hashing +{ + public enum HashAlg + { + SHA512 = 64, + SHA384 = 48, + SHA256 = 32, + SHA1 = 20, + MD5 = 16 + } + + /// <summary> + /// The binary hash encoding type + /// </summary> + public enum HashEncodingMode + { + /// <summary> + /// Specifies the Base64 character encoding + /// </summary> + Base64 = 64, + /// <summary> + /// Specifies the hexadecimal character encoding + /// </summary> + Hexadecimal = 16, + /// <summary> + /// Specifies the Base32 character encoding + /// </summary> + Base32 = 32 + } + + /// <summary> + /// Provides simple methods for common managed hashing functions + /// </summary> + public static partial class ManagedHash + { + private static readonly Encoding CharEncoding = Encoding.UTF8; + + /// <summary> + /// Uses the UTF8 character encoding to encode the string, then + /// attempts to compute the hash and store the results into the output buffer + /// </summary> + /// <param name="data">String to hash</param> + /// <param name="buffer">The hash output buffer</param> + /// <param name="type">The hash algorithm to use</param> + /// <returns>The number of bytes written to the buffer, false if the hash could not be computed</returns> + /// <exception cref="ArgumentException"></exception> + public static ERRNO ComputeHash(ReadOnlySpan<char> data, Span<byte> buffer, HashAlg type) + { + int byteCount = CharEncoding.GetByteCount(data); + //Alloc buffer + using UnsafeMemoryHandle<byte> binbuf = Memory.UnsafeAlloc<byte>(byteCount, true); + //Encode data + byteCount = CharEncoding.GetBytes(data, binbuf); + //hash the buffer + return ComputeHash(binbuf.Span[..byteCount], buffer, type); + } + + /// <summary> + /// Uses the UTF8 character encoding to encode the string, then + /// attempts to compute the hash and store the results into the output buffer + /// </summary> + /// <param name="data">String to hash</param> + /// <param name="type">The hash algorithm to use</param> + /// <returns>The number of bytes written to the buffer, false if the hash could not be computed</returns> + /// <exception cref="ArgumentException"></exception> + public static byte[] ComputeHash(ReadOnlySpan<char> data, HashAlg type) + { + int byteCount = CharEncoding.GetByteCount(data); + //Alloc buffer + using UnsafeMemoryHandle<byte> binbuf = Memory.UnsafeAlloc<byte>(byteCount, true); + //Encode data + byteCount = CharEncoding.GetBytes(data, binbuf); + //hash the buffer + return ComputeHash(binbuf.Span[..byteCount], type); + } + + /// <summary> + /// Hashes the data parameter to the output buffer using the specified algorithm type + /// </summary> + /// <param name="data">String to hash</param> + /// <param name="output">The hash output buffer</param> + /// <param name="type">The hash algorithm to use</param> + /// <returns>The number of bytes written to the buffer, <see cref="ERRNO.E_FAIL"/> if the hash could not be computed</returns> + /// <exception cref="ArgumentException"></exception> + public static ERRNO ComputeHash(ReadOnlySpan<byte> data, Span<byte> output, HashAlg type) + { + //hash the buffer + return type switch + { + HashAlg.SHA512 => SHA512.TryHashData(data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.SHA384 => SHA384.TryHashData(data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.SHA256 => SHA256.TryHashData(data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.SHA1 => SHA1.TryHashData(data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.MD5 => MD5.TryHashData(data, output, out int count) ? count : ERRNO.E_FAIL, + _ => throw new ArgumentException("Hash algorithm is not supported"), + }; + } + + /// <summary> + /// Hashes the data parameter to the output buffer using the specified algorithm type + /// </summary> + /// <param name="data">String to hash</param> + /// <param name="type">The hash algorithm to use</param> + /// <returns>A byte array that contains the hash of the data buffer</returns> + /// <exception cref="ArgumentException"></exception> + public static byte[] ComputeHash(ReadOnlySpan<byte> data, HashAlg type) + { + //hash the buffer + return type switch + { + HashAlg.SHA512 => SHA512.HashData(data), + HashAlg.SHA384 => SHA384.HashData(data), + HashAlg.SHA256 => SHA256.HashData(data), + HashAlg.SHA1 => SHA1.HashData(data), + HashAlg.MD5 => MD5.HashData(data), + _ => throw new ArgumentException("Hash algorithm is not supported"), + }; + } + + /// <summary> + /// Hashes the data parameter to the output buffer using the specified algorithm type + /// </summary> + /// <param name="data">String to hash</param> + /// <param name="type">The hash algorithm to use</param> + /// <param name="mode">The data encoding mode</param> + /// <returns>The encoded hash of the input data</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="CryptographicException"></exception> + public static string ComputeHash(ReadOnlySpan<byte> data, HashAlg type, HashEncodingMode mode) + { + //Alloc hash buffer + Span<byte> hashBuffer = stackalloc byte[(int)type]; + //hash the buffer + ERRNO count = ComputeHash(data, hashBuffer, type); + if (!count) + { + throw new CryptographicException("Failed to compute the hash of the data"); + } + //Convert to hex string + return mode switch + { + HashEncodingMode.Hexadecimal => Convert.ToHexString(hashBuffer.Slice(0, count)), + HashEncodingMode.Base64 => Convert.ToBase64String(hashBuffer.Slice(0, count)), + HashEncodingMode.Base32 => VnEncoding.ToBase32String(hashBuffer.Slice(0, count)), + _ => throw new ArgumentException("Encoding mode is not supported"), + }; + } + + /// <summary> + /// Uses the UTF8 character encoding to encode the string, then computes the hash and encodes + /// the hash to the specified encoding + /// </summary> + /// <param name="data">String to hash</param> + /// <param name="type">The hash algorithm to use</param> + /// <param name="mode">The data encoding mode</param> + /// <returns>The encoded hash of the input data</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="CryptographicException"></exception> + public static string ComputeHash(ReadOnlySpan<char> data, HashAlg type, HashEncodingMode mode) + { + //Alloc hash buffer + Span<byte> hashBuffer = stackalloc byte[(int)type]; + //hash the buffer + ERRNO count = ComputeHash(data, hashBuffer, type); + if (!count) + { + throw new CryptographicException("Failed to compute the hash of the data"); + } + //Convert to hex string + return mode switch + { + HashEncodingMode.Hexadecimal => Convert.ToHexString(hashBuffer.Slice(0, count)), + HashEncodingMode.Base64 => Convert.ToBase64String(hashBuffer.Slice(0, count)), + HashEncodingMode.Base32 => VnEncoding.ToBase32String(hashBuffer.Slice(0, count)), + _ => throw new ArgumentException("Encoding mode is not supported"), + }; + } + + + public static string ComputeHexHash(ReadOnlySpan<byte> data, HashAlg type) => ComputeHash(data, type, HashEncodingMode.Hexadecimal); + public static string ComputeBase64Hash(ReadOnlySpan<byte> data, HashAlg type) => ComputeHash(data, type, HashEncodingMode.Base64); + public static string ComputeHexHash(ReadOnlySpan<char> data, HashAlg type) => ComputeHash(data, type, HashEncodingMode.Hexadecimal); + public static string ComputeBase64Hash(ReadOnlySpan<char> data, HashAlg type) => ComputeHash(data, type, HashEncodingMode.Base64); + + /// <summary> + /// Computes the HMAC of the specified character buffer using the specified key and + /// writes the resuts to the output buffer. + /// </summary> + /// <param name="key">The HMAC key</param> + /// <param name="data">The character buffer to compute the encoded HMAC of</param> + /// <param name="output">The buffer to write the hash to</param> + /// <param name="type">The <see cref="HashAlg"/> type used to compute the HMAC</param> + /// <returns>The number of bytes written to the ouput buffer or <see cref="ERRNO.E_FAIL"/> if the operation failed</returns> + /// <exception cref="ArgumentException"></exception> + public static ERRNO ComputeHmac(ReadOnlySpan<byte> key, ReadOnlySpan<char> data, Span<byte> output, HashAlg type) + { + int byteCount = CharEncoding.GetByteCount(data); + //Alloc buffer + using UnsafeMemoryHandle<byte> binbuf = Memory.UnsafeAlloc<byte>(byteCount, true); + //Encode data + byteCount = CharEncoding.GetBytes(data, binbuf); + //hash the buffer + return ComputeHmac(key, binbuf.Span[..byteCount], output, type); + } + + /// <summary> + /// Computes the HMAC of the specified character buffer using the specified key and + /// writes the resuts to a new buffer to return + /// </summary> + /// <param name="key">The HMAC key</param> + /// <param name="data">The data buffer to compute the HMAC of</param> + /// <param name="type">The <see cref="HashAlg"/> type used to compute the HMAC</param> + /// <returns>A buffer containg the computed HMAC</returns> + /// <exception cref="ArgumentException"></exception> + public static byte[] ComputeHmac(ReadOnlySpan<byte> key, ReadOnlySpan<char> data, HashAlg type) + { + int byteCount = CharEncoding.GetByteCount(data); + //Alloc buffer + using UnsafeMemoryHandle<byte> binbuf = Memory.UnsafeAlloc<byte>(byteCount, true); + //Encode data + byteCount = CharEncoding.GetBytes(data, binbuf); + //hash the buffer + return ComputeHmac(key, binbuf.Span[..byteCount], type); + } + /// <summary> + /// Computes the HMAC of the specified data buffer using the specified key and + /// writes the resuts to the output buffer. + /// </summary> + /// <param name="key">The HMAC key</param> + /// <param name="data">The data buffer to compute the HMAC of</param> + /// <param name="output">The buffer to write the hash to</param> + /// <param name="type">The <see cref="HashAlg"/> type used to compute the HMAC</param> + /// <returns>The number of bytes written to the ouput buffer or <see cref="ERRNO.E_FAIL"/> if the operation failed</returns> + /// <exception cref="ArgumentException"></exception> + public static ERRNO ComputeHmac(ReadOnlySpan<byte> key, ReadOnlySpan<byte> data, Span<byte> output, HashAlg type) + { + //hash the buffer + return type switch + { + HashAlg.SHA512 => HMACSHA512.TryHashData(key, data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.SHA384 => HMACSHA384.TryHashData(key, data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.SHA256 => HMACSHA256.TryHashData(key, data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.SHA1 => HMACSHA1.TryHashData(key, data, output, out int count) ? count : ERRNO.E_FAIL, + HashAlg.MD5 => HMACMD5.TryHashData(key, data, output, out int count) ? count : ERRNO.E_FAIL, + _ => throw new ArgumentException("Hash algorithm is not supported"), + }; + } + + /// <summary> + /// Computes the HMAC of the specified data buffer using the specified key and + /// writes the resuts to a new buffer to return + /// </summary> + /// <param name="key">The HMAC key</param> + /// <param name="data">The data buffer to compute the HMAC of</param> + /// <param name="type">The <see cref="HashAlg"/> type used to compute the HMAC</param> + /// <returns>A buffer containg the computed HMAC</returns> + /// <exception cref="ArgumentException"></exception> + public static byte[] ComputeHmac(ReadOnlySpan<byte> key, ReadOnlySpan<byte> data, HashAlg type) + { + //hash the buffer + return type switch + { + HashAlg.SHA512 => HMACSHA512.HashData(key, data), + HashAlg.SHA384 => HMACSHA384.HashData(key, data), + HashAlg.SHA256 => HMACSHA256.HashData(key, data), + HashAlg.SHA1 => HMACSHA1.HashData(key, data), + HashAlg.MD5 => HMACMD5.HashData(key, data), + _ => throw new ArgumentException("Hash algorithm is not supported"), + }; + } + + /// <summary> + /// Computes the HMAC of the specified data buffer and encodes the result in + /// the specified <see cref="HashEncodingMode"/> + /// </summary> + /// <param name="key">The HMAC key</param> + /// <param name="data">The data buffer to compute the HMAC of</param> + /// <param name="type">The <see cref="HashAlg"/> type used to compute the HMAC</param> + /// <param name="mode">The encoding type for the output data</param> + /// <returns>The encoded string of the result</returns> + /// <exception cref="ArgumentException"></exception> + public static string ComputeHmac(ReadOnlySpan<byte> key, ReadOnlySpan<byte> data, HashAlg type, HashEncodingMode mode) + { + //Alloc hash buffer + Span<byte> hashBuffer = stackalloc byte[(int)type]; + //hash the buffer + ERRNO count = ComputeHmac(key, data, hashBuffer, type); + if (!count) + { + throw new InternalBufferTooSmallException("Failed to compute the hash of the data"); + } + //Convert to hex string + return mode switch + { + HashEncodingMode.Hexadecimal => Convert.ToHexString(hashBuffer.Slice(0, count)), + HashEncodingMode.Base64 => Convert.ToBase64String(hashBuffer.Slice(0, count)), + HashEncodingMode.Base32 => VnEncoding.ToBase32String(hashBuffer.Slice(0, count)), + _ => throw new ArgumentException("Encoding mode is not supported"), + }; + } + + /// <summary> + /// Computes the HMAC of the specified data buffer and encodes the result in + /// the specified <see cref="HashEncodingMode"/> + /// </summary> + /// <param name="key">The HMAC key</param> + /// <param name="data">The character buffer to compute the HMAC of</param> + /// <param name="type">The <see cref="HashAlg"/> type used to compute the HMAC</param> + /// <param name="mode">The encoding type for the output data</param> + /// <returns>The encoded string of the result</returns> + /// <exception cref="ArgumentException"></exception> + public static string ComputeHmac(ReadOnlySpan<byte> key, ReadOnlySpan<char> data, HashAlg type, HashEncodingMode mode) + { + //Alloc hash buffer + Span<byte> hashBuffer = stackalloc byte[(int)type]; + //hash the buffer + ERRNO count = ComputeHmac(key, data, hashBuffer, type); + if (!count) + { + throw new InternalBufferTooSmallException("Failed to compute the hash of the data"); + } + //Convert to hex string + return mode switch + { + HashEncodingMode.Hexadecimal => Convert.ToHexString(hashBuffer.Slice(0, count)), + HashEncodingMode.Base64 => Convert.ToBase64String(hashBuffer.Slice(0, count)), + HashEncodingMode.Base32 => VnEncoding.ToBase32String(hashBuffer.Slice(0, count)), + _ => throw new ArgumentException("Encoding mode is not supported"), + }; + } + } +} diff --git a/Hashing.Portable/src/RandomHash.cs b/Hashing.Portable/src/RandomHash.cs new file mode 100644 index 0000000..5a4fc66 --- /dev/null +++ b/Hashing.Portable/src/RandomHash.cs @@ -0,0 +1,149 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Hashing.Portable +* File: RandomHash.cs +* +* RandomHash.cs is part of VNLib.Hashing.Portable which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Hashing.Portable is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Hashing.Portable is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Hashing.Portable. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.Memory; + +namespace VNLib.Hashing +{ + /// <summary> + /// Produces random cryptographic data in common formats + /// </summary> + public static class RandomHash + { + + /// <summary> + /// Generates a cryptographic random number, computes the hash, and encodes the hash as a string. + /// </summary> + /// <param name="alg">The hash algorithm to use when computing the hash</param> + /// <param name="size">Number of random bytes</param> + /// <param name="encoding"></param> + /// <returns>String containing hash of the random number</returns> + public static string GetRandomHash(HashAlg alg, int size = 64, HashEncodingMode encoding = HashEncodingMode.Base64) + { + //Get temporary buffer for storing random keys + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(size); + //Fill with random non-zero bytes + GetRandomBytes(buffer.Span); + //Compute hash + return ManagedHash.ComputeHash(buffer.Span, alg, encoding); + } + + /// <summary> + /// Gets the sha512 hash of a new GUID + /// </summary> + /// <returns>String containing hash of the GUID</returns> + /// <exception cref="FormatException"></exception> + public static string GetGuidHash(HashAlg alg, HashEncodingMode encoding = HashEncodingMode.Base64) + { + //Get temp buffer + Span<byte> buffer = stackalloc byte[16]; + //Get a new GUID and write bytes to + if (!Guid.NewGuid().TryWriteBytes(buffer)) + { + throw new FormatException("Failed to get a guid hash"); + } + return ManagedHash.ComputeHash(buffer, alg, encoding); + } + /// <summary> + /// Generates a secure random number and seeds a GUID object, then returns the string GUID + /// </summary> + /// <returns>Guid string</returns> + public static Guid GetSecureGuid() + { + //Get temp buffer + Span<byte> buffer = stackalloc byte[16]; + //Generate non zero bytes + GetRandomBytes(buffer); + //Get a GUID initialized with the key data and return the string represendation + return new Guid(buffer); + } + + /// <summary> + /// Generates a cryptographic random number and returns the base64 string of that number + /// </summary> + /// <param name="size">Number of random bytes</param> + /// <returns>Base64 string of the random number</returns> + public static string GetRandomBase64(int size = 64) + { + //Get temp buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(size); + //Generate non zero bytes + GetRandomBytes(buffer.Span); + //Convert to base 64 + return Convert.ToBase64String(buffer.Span, Base64FormattingOptions.None); + } + /// <summary> + /// Generates a cryptographic random number and returns the hex string of that number + /// </summary> + /// <param name="size">Number of random bytes</param> + /// <returns>Hex string of the random number</returns> + public static string GetRandomHex(int size = 64) + { + //Get temp buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(size); + //Generate non zero bytes + GetRandomBytes(buffer.Span); + //Convert to hex + return Convert.ToHexString(buffer.Span); + } + /// <summary> + /// Generates a cryptographic random number and returns the Base32 encoded string of that number + /// </summary> + /// <param name="size">Number of random bytes</param> + /// <returns>Base32 string of the random number</returns> + public static string GetRandomBase32(int size = 64) + { + //Get temporary buffer for storing random keys + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(size); + //Fill with random non-zero bytes + GetRandomBytes(buffer.Span); + //Return string of encoded data + return VnEncoding.ToBase32String(buffer.Span); + } + + /// <summary> + /// Allocates a new byte[] of the specified size and fills it with non-zero random values + /// </summary> + /// <param name="size">Number of random bytes</param> + /// <returns>byte[] containing the random data</returns> + public static byte[] GetRandomBytes(int size = 64) + { + byte[] rand = new byte[size]; + GetRandomBytes(rand); + return rand; + } + /// <summary> + /// Fill the buffer with non-zero bytes + /// </summary> + /// <param name="data">Buffer to fill</param> + public static void GetRandomBytes(Span<byte> data) + { + RandomNumberGenerator.Fill(data); + } + } +}
\ No newline at end of file diff --git a/Hashing.Portable/src/VNLib.Hashing.Portable.csproj b/Hashing.Portable/src/VNLib.Hashing.Portable.csproj new file mode 100644 index 0000000..9494521 --- /dev/null +++ b/Hashing.Portable/src/VNLib.Hashing.Portable.csproj @@ -0,0 +1,51 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <Version>1.0.1.3</Version> + <Product>VNLib Hashing Function/Alg Library</Product> + <Description>Provides managed and random cryptocraphic hashing helper classes, including complete Argon2 password hashing.</Description> + <Authors>Vaughn Nugent</Authors> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <AssemblyName>VNLib.Hashing.Portable</AssemblyName> + <RootNamespace>VNLib.Hashing</RootNamespace> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <!-- Resolve nuget dll files and store them in the output dir --> + <PropertyGroup> + <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> + <Nullable>enable</Nullable> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <AnalysisLevel>latest-all</AnalysisLevel> + <ProduceReferenceAssembly>True</ProduceReferenceAssembly> + <AllowUnsafeBlocks>True</AllowUnsafeBlocks> + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + </ItemGroup> + + <ItemGroup> + <None Update="Argon2.dll"> + <CopyToOutputDirectory>Never</CopyToOutputDirectory> + </None> + <None Update="Argon2RefDll.dll"> + <CopyToOutputDirectory>Never</CopyToOutputDirectory> + </None> + </ItemGroup> + +</Project> diff --git a/Net.Http/LICENSE.txt b/Net.Http/LICENSE.txt new file mode 100644 index 0000000..147bcd6 --- /dev/null +++ b/Net.Http/LICENSE.txt @@ -0,0 +1,195 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU Affero GPL version 3.0 (or any later version). + +GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/> +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. +Preamble + +The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. + +A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. + +The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. + +An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. + +The precise terms and conditions for copying, distribution and modification follow. +TERMS AND CONDITIONS +0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public License. + +"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based on the Program. + +To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. + +An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. +1. Source Code. + +The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. + +A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same work. +2. Basic Permissions. + +All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. +3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. + +When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. +4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. +5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified it, and giving a relevant date. + b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". + c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. + d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. + +A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. +6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: + + a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. + b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. + c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. + d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. + e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. + +"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. + +If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). + +The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. + +Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. +7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or + b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or + c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or + d) Limiting the use for publicity purposes of names of licensors or authors of the material; or + e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or + f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. + +All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. +8. Termination. + +You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). + +However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. + +Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. + +Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. +9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. +10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. +11. Patents. + +A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. + +If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. + +A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. +12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. +13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. +14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. + +Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. +15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. +16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Net.Http/readme.md b/Net.Http/readme.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/Net.Http/readme.md diff --git a/Net.Http/src/AlternateProtocolBase.cs b/Net.Http/src/AlternateProtocolBase.cs new file mode 100644 index 0000000..929bc33 --- /dev/null +++ b/Net.Http/src/AlternateProtocolBase.cs @@ -0,0 +1,96 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: AlternateProtocolBase.cs +* +* AlternateProtocolBase.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Net.Http.Core; + +namespace VNLib.Net.Http +{ + /// <summary> + /// A base class for all non-http protocol handlers + /// </summary> + public abstract class AlternateProtocolBase : MarshalByRefObject, IAlternateProtocol + { + /// <summary> + /// A cancelation source that allows for canceling running tasks, that is linked + /// to the server that called <see cref="RunAsync(Stream)"/>. + /// </summary> + /// <remarks> + /// This property is only available while the <see cref="RunAsync(Stream)"/> + /// method is executing + /// </remarks> +#pragma warning disable CS8618 // Non-nullable field must contain a non-null value when exiting constructor. Consider declaring as nullable. + protected CancellationTokenSource CancelSource { get; private set; } +#pragma warning restore CS8618 // Non-nullable field must contain a non-null value when exiting constructor. Consider declaring as nullable. + + ///<inheritdoc/> + async Task IAlternateProtocol.RunAsync(Stream transport, CancellationToken handlerToken) + { + //Create new cancel source + CancelSource ??= new(); + //Register the token to cancel the source and save the registration for unregister on dispose + CancellationTokenRegistration Registration = handlerToken.Register(CancelSource.Cancel); + try + { + //Call child initialize method + await RunAsync(new AlternateProtocolTransportStreamWrapper(transport)); + CancelSource.Cancel(); + } + finally + { + //dispose the cancelation registration + await Registration.DisposeAsync(); + //Dispose cancel source + CancelSource.Dispose(); + } + } + + /// <summary> + /// Is the current socket connected using transport security + /// </summary> + public virtual bool IsSecure { get; init; } + + /// <summary> + /// Determines if the instance is pending cancelation + /// </summary> + public bool IsCancellationRequested => CancelSource.IsCancellationRequested; + + /// <summary> + /// Cancels all pending operations. This session will be unusable after this function is called + /// </summary> + public virtual void CancelAll() => CancelSource?.Cancel(); + + /// <summary> + /// Called when the protocol swtich handshake has completed and the transport is + /// available for the new protocol + /// </summary> + /// <param name="transport">The transport stream</param> + /// <returns>A task that represents the active use of the transport, and when complete all operations are unwound</returns> + protected abstract Task RunAsync(Stream transport); + } +}
\ No newline at end of file diff --git a/Net.Http/src/ConnectionInfo.cs b/Net.Http/src/ConnectionInfo.cs new file mode 100644 index 0000000..6e1660d --- /dev/null +++ b/Net.Http/src/ConnectionInfo.cs @@ -0,0 +1,166 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ConnectionInfo.cs +* +* ConnectionInfo.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Linq; +using System.Text; +using System.Collections.Generic; +using System.Security.Authentication; + +using VNLib.Net.Http.Core; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http +{ + ///<inheritdoc/> + internal sealed class ConnectionInfo : IConnectionInfo + { + private HttpContext Context; + + ///<inheritdoc/> + public Uri RequestUri => Context.Request.Location; + ///<inheritdoc/> + public string Path => RequestUri.LocalPath; + ///<inheritdoc/> + public string? UserAgent => Context.Request.UserAgent; + ///<inheritdoc/> + public IHeaderCollection Headers { get; private set; } + ///<inheritdoc/> + public bool CrossOrigin { get; } + ///<inheritdoc/> + public bool IsWebSocketRequest { get; } + ///<inheritdoc/> + public ContentType ContentType => Context.Request.ContentType; + ///<inheritdoc/> + public HttpMethod Method => Context.Request.Method; + ///<inheritdoc/> + public HttpVersion ProtocolVersion => Context.Request.HttpVersion; + ///<inheritdoc/> + public bool IsSecure => Context.Request.EncryptionVersion != SslProtocols.None; + ///<inheritdoc/> + public SslProtocols SecurityProtocol => Context.Request.EncryptionVersion; + ///<inheritdoc/> + public Uri? Origin => Context.Request.Origin; + ///<inheritdoc/> + public Uri? Referer => Context.Request.Referrer; + ///<inheritdoc/> + public Tuple<long, long>? Range => Context.Request.Range; + ///<inheritdoc/> + public IPEndPoint LocalEndpoint => Context.Request.LocalEndPoint; + ///<inheritdoc/> + public IPEndPoint RemoteEndpoint => Context.Request.RemoteEndPoint; + ///<inheritdoc/> + public Encoding Encoding => Context.ParentServer.Config.HttpEncoding; + ///<inheritdoc/> + public IReadOnlyDictionary<string, string> RequestCookies => Context.Request.Cookies; + ///<inheritdoc/> + public IEnumerable<string> Accept => Context.Request.Accept; + ///<inheritdoc/> + public TransportSecurityInfo? TransportSecurity => Context.GetSecurityInfo(); + + ///<inheritdoc/> + public bool Accepts(ContentType type) + { + //Get the content type string from he specified content type + string contentType = HttpHelpers.GetContentTypeString(type); + return Accepts(contentType); + } + ///<inheritdoc/> + public bool Accepts(string contentType) + { + if (AcceptsAny()) + { + return true; + } + + //If client accepts exact requested encoding + if (Accept.Contains(contentType)) + { + return true; + } + + //Search accept types to determine if the content type is acceptable + bool accepted = Accept + .Where(ctype => + { + //Get prinary side of mime type + ReadOnlySpan<char> primary = contentType.AsSpan().SliceBeforeParam('/'); + ReadOnlySpan<char> ctSubType = ctype.AsSpan().SliceBeforeParam('/'); + //See if accepts any subtype, or the primary sub-type matches + return ctSubType[0] == '*' || ctSubType.Equals(primary, StringComparison.OrdinalIgnoreCase); + }).Any(); + return accepted; + } + /// <summary> + /// Determines if the connection accepts any content type + /// </summary> + /// <returns>true if the connection accepts any content typ, false otherwise</returns> + private bool AcceptsAny() + { + //Accept any if no accept header was present, or accept all value */* + return Context.Request.Accept.Count == 0 || Accept.Where(static t => t.StartsWith("*/*", StringComparison.OrdinalIgnoreCase)).Any(); + } + ///<inheritdoc/> + public void SetCookie(string name, string value, string? domain, string? path, TimeSpan Expires, CookieSameSite sameSite, bool httpOnly, bool secure) + { + //Create the new cookie + HttpCookie cookie = new(name) + { + Value = value, + Domain = domain, + Path = path, + MaxAge = Expires, + //Set the session lifetime flag if the timeout is max value + IsSession = Expires == TimeSpan.MaxValue, + //If the connection is cross origin, then we need to modify the secure and samsite values + SameSite = CrossOrigin ? CookieSameSite.None : sameSite, + Secure = secure | CrossOrigin, + HttpOnly = httpOnly + }; + //Set the cookie + Context.Response.AddCookie(cookie); + } + + internal ConnectionInfo(HttpContext ctx) + { + //Create new header collection + Headers = new VnHeaderCollection(ctx); + //set co value + CrossOrigin = ctx.Request.IsCrossOrigin(); + //Set websocket status + IsWebSocketRequest = ctx.Request.IsWebSocketRequest(); + //Update the context referrence + Context = ctx; + } + +#nullable disable + internal void Clear() + { + Context = null; + (Headers as VnHeaderCollection).Clear(); + Headers = null; + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/HttpContext.cs b/Net.Http/src/Core/HttpContext.cs new file mode 100644 index 0000000..43d1975 --- /dev/null +++ b/Net.Http/src/Core/HttpContext.cs @@ -0,0 +1,170 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpContext.cs +* +* HttpContext.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Runtime.CompilerServices; + +using VNLib.Utils; +using VNLib.Utils.Memory.Caching; + + +namespace VNLib.Net.Http.Core +{ + internal sealed partial class HttpContext : IConnectionContext, IReusable + { + /// <summary> + /// When set as a response flag, disables response compression for + /// the current request/response flow + /// </summary> + public const ulong COMPRESSION_DISABLED_MSK = 0x01UL; + + /// <summary> + /// The reusable http request container + /// </summary> + public readonly HttpRequest Request; + /// <summary> + /// The reusable response controler + /// </summary> + public readonly HttpResponse Response; + /// <summary> + /// The http server that this context is bound to + /// </summary> + public readonly HttpServer ParentServer; + /// <summary> + /// The shared transport header reader buffer + /// </summary> + public readonly SharedHeaderReaderBuffer RequestBuffer; + + /// <summary> + /// The response entity body container + /// </summary> + public readonly IHttpResponseBody ResponseBody; + + /// <summary> + /// A collection of flags that can be used to control the way the context + /// responds to client requests + /// </summary> + public readonly BitField ContextFlags; + + /// <summary> + /// Gets or sets the alternate application protocol to swtich to + /// </summary> + public IAlternateProtocol? AlternateProtocol { get; set; } + + private readonly ResponseWriter responseWriter; + private ITransportContext? _ctx; + + public HttpContext(HttpServer server) + { + /* + * Local method for retreiving the transport stream, + * this adds protection/debug from response/request + * containers not allowed to maintain referrences + * to a transport stream after it has been released + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + Stream GetStream() => _ctx!.ConnectionStream; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + HttpVersion GetVersion() => Request.HttpVersion; + + ParentServer = server; + + //Create new request + Request = new HttpRequest(GetStream); + + //create a new response object + Response = new HttpResponse( + server.Config.HttpEncoding, + ParentServer.Config.ResponseHeaderBufferSize, + ParentServer.Config.ChunkedResponseAccumulatorSize, + GetStream, + GetVersion); + + //The shared request parsing buffer + RequestBuffer = new(server.Config.HeaderBufferSize); + + //Init response writer + ResponseBody = responseWriter = new ResponseWriter(); + + ContextFlags = new(0); + } + + public TransportSecurityInfo? GetSecurityInfo() => _ctx?.GetSecurityInfo(); + + + #region LifeCycle Hooks + + ///<inheritdoc/> + public void InitializeContext(ITransportContext ctx) => _ctx = ctx; + + ///<inheritdoc/> + public void BeginRequest() + { + //Clear all flags + ContextFlags.ClearAll(); + + //Lifecycle on new request + Request.OnNewRequest(); + Response.OnNewRequest(); + RequestBuffer.OnNewRequest(); + + //Initialize the request + Request.Initialize(_ctx!, ParentServer.Config.DefaultHttpVersion); + } + + ///<inheritdoc/> + public void EndRequest() + { + AlternateProtocol = null; + + Request.OnComplete(); + Response.OnComplete(); + RequestBuffer.OnComplete(); + responseWriter.OnComplete(); + } + + void IReusable.Prepare() + { + Request.OnPrepare(); + Response.OnPrepare(); + RequestBuffer.OnPrepare(); + } + + bool IReusable.Release() + { + _ctx = null; + + //Release response/requqests + Request.OnRelease(); + Response.OnRelease(); + RequestBuffer.OnRelease(); + + return true; + } + + #endregion + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/HttpCookie.cs b/Net.Http/src/Core/HttpCookie.cs new file mode 100644 index 0000000..f5408b2 --- /dev/null +++ b/Net.Http/src/Core/HttpCookie.cs @@ -0,0 +1,125 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpCookie.cs +* +* HttpCookie.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http.Core +{ + internal class HttpCookie : IStringSerializeable, IEquatable<HttpCookie> + { + public string Name { get; } + public string? Value { get; init; } + public string? Domain { get; init; } + public string? Path { get; init; } + public TimeSpan MaxAge { get; init; } + public CookieSameSite SameSite { get; init; } + public bool Secure { get; init; } + public bool HttpOnly { get; init; } + public bool IsSession { get; init; } + + public HttpCookie(string name) + { + this.Name = name; + } + + public string Compile() + { + throw new NotImplementedException(); + } + public void Compile(ref ForwardOnlyWriter<char> writer) + { + //set the name of the cookie + writer.Append(Name); + writer.Append('='); + //set name + writer.Append(Value); + //Only set the max age parameter if the cookie is not a session cookie + if (!IsSession) + { + writer.Append("; Max-Age="); + writer.Append((int)MaxAge.TotalSeconds); + } + //Make sure domain is set + if (!string.IsNullOrWhiteSpace(Domain)) + { + writer.Append("; Domain="); + writer.Append(Domain); + } + //Check and set path + if (!string.IsNullOrWhiteSpace(Path)) + { + //Set path + writer.Append("; Path="); + writer.Append(Path); + } + writer.Append("; SameSite="); + //Set the samesite flag based on the enum value + switch (SameSite) + { + case CookieSameSite.None: + writer.Append("None"); + break; + case CookieSameSite.SameSite: + writer.Append("Strict"); + break; + case CookieSameSite.Lax: + default: + writer.Append("Lax"); + break; + } + //Set httponly flag + if (HttpOnly) + { + writer.Append("; HttpOnly"); + } + //Set secure flag + if (Secure) + { + writer.Append("; Secure"); + } + } + public ERRNO Compile(in Span<char> buffer) + { + ForwardOnlyWriter<char> writer = new(buffer); + Compile(ref writer); + return writer.Written; + } + + public override int GetHashCode() => Name.GetHashCode(); + + public override bool Equals(object? obj) + { + return obj is HttpCookie other && Equals(other); + } + + public bool Equals(HttpCookie? other) + { + return other != null && Name.Equals(other.Name, StringComparison.OrdinalIgnoreCase); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/HttpEvent.cs b/Net.Http/src/Core/HttpEvent.cs new file mode 100644 index 0000000..7d7c1e7 --- /dev/null +++ b/Net.Http/src/Core/HttpEvent.cs @@ -0,0 +1,141 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpEvent.cs +* +* HttpEvent.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Collections.Generic; +using System.Runtime.CompilerServices; + +using VNLib.Net.Http.Core; + +namespace VNLib.Net.Http +{ + internal sealed class HttpEvent : MarshalByRefObject, IHttpEvent + { + private HttpContext Context; + private ConnectionInfo _ci; + + internal HttpEvent(HttpContext ctx) + { + Context = ctx; + _ci = new ConnectionInfo(ctx); + } + + ///<inheritdoc/> + IConnectionInfo IHttpEvent.Server => _ci; + + ///<inheritdoc/> + HttpServer IHttpEvent.OriginServer => Context.ParentServer; + + ///<inheritdoc/> + IReadOnlyDictionary<string, string> IHttpEvent.QueryArgs => Context.Request.RequestBody.QueryArgs; + ///<inheritdoc/> + IReadOnlyDictionary<string, string> IHttpEvent.RequestArgs => Context.Request.RequestBody.RequestArgs; + ///<inheritdoc/> + IReadOnlyList<FileUpload> IHttpEvent.Files => Context.Request.RequestBody.Uploads; + + ///<inheritdoc/> + void IHttpEvent.DisableCompression() => Context.ContextFlags.Set(HttpContext.COMPRESSION_DISABLED_MSK); + + ///<inheritdoc/> + void IHttpEvent.DangerousChangeProtocol(IAlternateProtocol protocolHandler) + { + if(Context.AlternateProtocol != null) + { + throw new InvalidOperationException("A protocol handler was already specified"); + } + + _ = protocolHandler ?? throw new ArgumentNullException(nameof(protocolHandler)); + + //Set 101 status code + Context.Respond(HttpStatusCode.SwitchingProtocols); + Context.AlternateProtocol = protocolHandler; + } + + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + void IHttpEvent.CloseResponse(HttpStatusCode code) => Context.Respond(code); + + ///<inheritdoc/> + void IHttpEvent.CloseResponse(HttpStatusCode code, ContentType type, Stream stream) + { + //Check if the stream is valid. We will need to read the stream, and we will also need to get the length property + if (!stream.CanSeek || !stream.CanRead) + { + throw new IOException("The stream.Length property must be available and the stream must be readable"); + } + + //If stream is empty, ignore it, the server will default to 0 content length and avoid overhead + if (stream.Length == 0) + { + return; + } + + //Set status code + Context.Response.SetStatusCode(code); + + //Finally store the stream input + if(!(Context.ResponseBody as ResponseWriter)!.TrySetResponseBody(stream)) + { + throw new InvalidOperationException("A response body has already been set"); + } + + //Set content type header after body + Context.Response.Headers[HttpResponseHeader.ContentType] = HttpHelpers.GetContentTypeString(type); + } + + ///<inheritdoc/> + void IHttpEvent.CloseResponse(HttpStatusCode code, ContentType type, IMemoryResponseReader entity) + { + //If stream is empty, ignore it, the server will default to 0 content length and avoid overhead + if (entity.Remaining == 0) + { + return; + } + + //Set status code + Context.Response.SetStatusCode(code); + + //Finally store the stream input + if (!(Context.ResponseBody as ResponseWriter)!.TrySetResponseBody(entity)) + { + throw new InvalidOperationException("A response body has already been set"); + } + + //Set content type header after body + Context.Response.Headers[HttpResponseHeader.ContentType] = HttpHelpers.GetContentTypeString(type); + } + +#pragma warning disable CS8625 // Cannot convert null literal to non-nullable reference type. + internal void Clear() + { + //Clean up referrence types and cleanable objects + Context = null; + _ci.Clear(); + _ci = null; + } +#pragma warning restore CS8625 // Cannot convert null literal to non-nullable reference type. + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/HttpServerBase.cs b/Net.Http/src/Core/HttpServerBase.cs new file mode 100644 index 0000000..3a50672 --- /dev/null +++ b/Net.Http/src/Core/HttpServerBase.cs @@ -0,0 +1,312 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpServerBase.cs +* +* HttpServerBase.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +/* + * This file is the base of the HTTP server class that provides + * consts, statics, fields, and properties of the HttpServer class. + * + * Processing of HTTP connections and entities is contained in the + * processing partial file. + * + * Processing is configured to be asynchronous, utilizing .NETs + * asynchronous compilation services. To facilitate this but continue + * to use object caching, reusable stores must be usable across threads + * to function safely with async programming practices. + */ + +using System; +using System.Linq; +using System.Threading; +using System.Net.Sockets; +using System.Threading.Tasks; +using System.Collections.Generic; +using System.Security.Authentication; + +using VNLib.Utils.Logging; +using VNLib.Utils.Memory.Caching; + +using VNLib.Net.Http.Core; + +namespace VNLib.Net.Http +{ + + /// <summary> + /// Provides a resource efficient, high performance, single library HTTP(s) server, + /// with extensable processors and transport providers. + /// This class cannot be inherited + /// </summary> + public sealed partial class HttpServer : ICacheHolder + { + /// <summary> + /// The host key that determines a "wildcard" host, meaning the + /// default connection handler when an incomming connection has + /// not specific route + /// </summary> + public const string WILDCARD_KEY = "*"; + + private readonly ITransportProvider Transport; + private readonly IReadOnlyDictionary<string, IWebRoot> ServerRoots; + + #region caches + /// <summary> + /// The cached HTTP1/1 keepalive timeout header value + /// </summary> + private readonly string KeepAliveTimeoutHeaderValue; + /// <summary> + /// Reusable store for obtaining <see cref="HttpContext"/> + /// </summary> + private readonly ObjectRental<HttpContext> ContextStore; + /// <summary> + /// The cached header-line termination value + /// </summary> + private readonly ReadOnlyMemory<byte> HeaderLineTermination; + #endregion + + /// <summary> + /// The <see cref="HttpConfig"/> for the current server + /// </summary> + public HttpConfig Config { get; } + + /// <summary> + /// Gets a value indicating whether the server is listening for connections + /// </summary> + public bool Running { get; private set; } + + private CancellationTokenSource? StopToken; + + /// <summary> + /// Creates a new <see cref="HttpServer"/> with the specified configration copy (using struct). + /// Immutable data structures are initialzed. + /// </summary> + /// <param name="config">The configuration used to create the instance</param> + /// <param name="transport">The transport provider to listen to connections from</param> + /// <param name="sites">A collection of <see cref="IWebRoot"/>s that route incomming connetctions</param> + /// <exception cref="ArgumentException"></exception> + public HttpServer(HttpConfig config, ITransportProvider transport, IEnumerable<IWebRoot> sites) + { + //Validate the configuration + ValidateConfig(in config); + + Config = config; + //Configure roots and their directories + ServerRoots = sites.ToDictionary(static r => r.Hostname, static tv => tv, StringComparer.OrdinalIgnoreCase); + //Compile and store the timeout keepalive header + KeepAliveTimeoutHeaderValue = $"timeout={(int)Config.ConnectionKeepAlive.TotalSeconds}"; + //Store termination for the current instance + HeaderLineTermination = config.HttpEncoding.GetBytes(HttpHelpers.CRLF); + //Create a new context store + ContextStore = ObjectRental.CreateReusable(() => new HttpContext(this)); + //Setup config copy with the internal http pool + Transport = transport; + } + + private static void ValidateConfig(in HttpConfig conf) + { + _ = conf.HttpEncoding ?? throw new ArgumentException("HttpEncoding cannot be null", nameof(conf)); + _ = conf.ServerLog ?? throw new ArgumentException("ServerLog cannot be null", nameof(conf)); + + if (conf.ActiveConnectionRecvTimeout < -1) + { + throw new ArgumentException("ActiveConnectionRecvTimeout cannot be less than -1", nameof(conf)); + } + + //Chunked data accumulator must be at least 64 bytes (arbinrary value) + if (conf.ChunkedResponseAccumulatorSize < 64 || conf.ChunkedResponseAccumulatorSize == int.MaxValue) + { + throw new ArgumentException("ChunkedResponseAccumulatorSize cannot be less than 64 bytes", nameof(conf)); + } + + if (conf.CompressionLimit < 0) + { + throw new ArgumentException("CompressionLimit cannot be less than 0, set to 0 to disable response compression", nameof(conf)); + } + + if (conf.ConnectionKeepAlive < TimeSpan.Zero) + { + throw new ArgumentException("ConnectionKeepAlive cannot be less than 0", nameof(conf)); + } + + if (conf.DefaultHttpVersion == HttpVersion.NotSupported) + { + throw new ArgumentException("DefaultHttpVersion cannot be NotSupported", nameof(conf)); + } + + if (conf.DiscardBufferSize < 64) + { + throw new ArgumentException("DiscardBufferSize cannot be less than 64 bytes", nameof(conf)); + } + + if (conf.FormDataBufferSize < 64) + { + throw new ArgumentException("FormDataBufferSize cannot be less than 64 bytes", nameof(conf)); + } + + if (conf.HeaderBufferSize < 128) + { + throw new ArgumentException("HeaderBufferSize cannot be less than 128 bytes", nameof(conf)); + } + + if (conf.MaxFormDataUploadSize < 0) + { + throw new ArgumentException("MaxFormDataUploadSize cannot be less than 0, set to 0 to disable form-data uploads", nameof(conf)); + } + + if (conf.MaxOpenConnections < 0) + { + throw new ArgumentException("MaxOpenConnections cannot be less than 0", nameof(conf)); + } + + if (conf.MaxRequestHeaderCount < 1) + { + throw new ArgumentException("MaxRequestHeaderCount cannot be less than 1", nameof(conf)); + } + + if (conf.MaxUploadSize < 0) + { + throw new ArgumentException("MaxUploadSize cannot be less than 0", nameof(conf)); + } + + if (conf.ResponseBufferSize < 64) + { + throw new ArgumentException("ResponseBufferSize cannot be less than 64 bytes", nameof(conf)); + } + + if (conf.ResponseHeaderBufferSize < 128) + { + throw new ArgumentException("ResponseHeaderBufferSize cannot be less than 128 bytes", nameof(conf)); + } + + if (conf.SendTimeout < 1) + { + throw new ArgumentException("SendTimeout cannot be less than 1 millisecond", nameof(conf)); + } + } + + /// <summary> + /// Begins listening for connections on configured interfaces for configured hostnames. + /// </summary> + /// <param name="token">A token used to stop listening for incomming connections and close all open websockets</param> + /// <returns>A task that resolves when the server has exited</returns> + /// <exception cref="SocketException"></exception> + /// <exception cref="ThreadStateException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public Task Start(CancellationToken token) + { + StopToken = CancellationTokenSource.CreateLinkedTokenSource(token); + //Start servers with the new token source + Transport.Start(token); + //Start the listen task + return Task.Run(ListenWorkerDoWork, token); + } + + /* + * An SslStream may throw a win32 exception with HRESULT 0x80090327 + * when processing a client certificate (I believe anyway) only + * an issue on some clients (browsers) + */ + + private const int UKNOWN_CERT_AUTH_HRESULT = unchecked((int)0x80090327); + + /// <summary> + /// An invlaid frame size may happen if data is recieved on an open socket + /// but does not contain valid SSL handshake data + /// </summary> + private const int INVALID_FRAME_HRESULT = unchecked((int)0x80131620); + + /* + * A worker task that listens for connections from the transport + */ + private async Task ListenWorkerDoWork() + { + //Set running flag + Running = true; + + Config.ServerLog.Information("HTTP server {hc} listening for connections", GetHashCode()); + //Listen for connections until canceled + while (true) + { + try + { + //Listen for new connection + ITransportContext ctx = await Transport.AcceptAsync(StopToken!.Token); + //Try to dispatch the recieved event + _ = DataReceivedAsync(ctx).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + //Closing, exit loop + break; + } + catch (AuthenticationException ae) + { + Config.ServerLog.Error(ae); + } + catch (Exception ex) + { + Config.ServerLog.Error(ex); + } + } + //Clear all caches + CacheHardClear(); + //Clear running flag + Running = false; + Config.ServerLog.Information("HTTP server {hc} exiting", GetHashCode()); + } + + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public void CacheClear() => ContextStore.CacheClear(); + + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + public void CacheHardClear() => ContextStore.CacheHardClear(); + + /// <summary> + /// Writes the specialized log for a socket exception + /// </summary> + /// <param name="se">The socket exception to log</param> + public void WriteSocketExecption(SocketException se) + { + //When clause guards nulls + switch (se.SocketErrorCode) + + { + //Ignore aborted messages + case SocketError.ConnectionAborted: + return; + case SocketError.ConnectionReset: + Config.ServerLog.Debug("Connecion reset by client"); + return; + case SocketError.TimedOut: + Config.ServerLog.Debug("Socket operation timed out"); + return; + default: + Config.ServerLog.Information(se); + break; + } + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/HttpServerProcessing.cs b/Net.Http/src/Core/HttpServerProcessing.cs new file mode 100644 index 0000000..881b66c --- /dev/null +++ b/Net.Http/src/Core/HttpServerProcessing.cs @@ -0,0 +1,387 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpServerProcessing.cs +* +* HttpServerProcessing.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Threading; +using System.Net.Sockets; +using System.Threading.Tasks; +using System.Runtime.CompilerServices; + +using VNLib.Utils; +using VNLib.Utils.Logging; +using VNLib.Net.Http.Core; + +namespace VNLib.Net.Http +{ + public sealed partial class HttpServer + { + + private int OpenConnectionCount; + + //Event handler method for processing incoming data events + private async Task DataReceivedAsync(ITransportContext transportContext) + { + //Increment open connection count + Interlocked.Increment(ref OpenConnectionCount); + + //Rent a new context object to reuse + HttpContext context = ContextStore.Rent(); + + try + { + //Set write timeout + transportContext.ConnectionStream.WriteTimeout = Config.SendTimeout; + + //Init stream + context.InitializeContext(transportContext); + + //Keep the transport open and listen for messages as long as keepalive is enabled + do + { + //Set rx timeout low for initial reading + transportContext.ConnectionStream.ReadTimeout = Config.ActiveConnectionRecvTimeout; + + //Process the request + ERRNO keepalive = await ProcessHttpEventAsync(transportContext, context); + + //If the connection is closed, we can return + if (!keepalive) + { + break; + } + + //Set inactive keeaplive timeout + transportContext.ConnectionStream.ReadTimeout = (int)Config.ConnectionKeepAlive.TotalMilliseconds; + + //"Peek" or wait for more data to begin another request (may throw timeout exception when timmed out) + await transportContext.ConnectionStream.ReadAsync(Memory<byte>.Empty, StopToken!.Token); + + } while (true); + } + //Catch wrapped socket exceptions + catch(IOException ioe) when(ioe.InnerException is SocketException se) + { + WriteSocketExecption(se); + } + catch(SocketException se) + { + WriteSocketExecption(se); + } + catch (OperationCanceledException oce) + { + Config.ServerLog.Debug("Failed to receive transport data within a timeout period {m}, connection closed", oce.Message); + } + catch(Exception ex) + { + Config.ServerLog.Error(ex); + } + + //Dec open connection count + Interlocked.Decrement(ref OpenConnectionCount); + + //Return context to store + ContextStore.Return(context); + + //Close the transport async + try + { + await transportContext.CloseConnectionAsync(); + } + catch(Exception ex) + { + Config.ServerLog.Error(ex); + } + } + + + /// <summary> + /// Main event handler for all incoming connections + /// </summary> + /// <param name="transportContext">The <see cref="ITransportContext"/> describing the incoming connection</param> + /// <param name="context">Reusable context object</param> + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + private async Task<ERRNO> ProcessHttpEventAsync(ITransportContext transportContext, HttpContext context) + { + //Prepare http context to process a new message + context.BeginRequest(); + + try + { + //Try to parse the http request (may throw exceptions, let them propagate to the transport layer) + int status = (int)ParseRequest(transportContext, context); + + //Check status code for socket error, if so, return false to close the connection + if (status >= 1000) + { + return false; + } +#if DEBUG + //Write debug request log + if (Config.RequestDebugLog != null) + { + Config.RequestDebugLog.Verbose(context.Request.ToString()); + } +#endif + //process the request + ERRNO keepalive = await ProcessRequestAsync(context, (HttpStatusCode)status); + //Store alternate protocol if set + IAlternateProtocol? alternateProtocol = context.AlternateProtocol; + //Close the response + await context.WriteResponseAsync(StopToken!.Token); + //See if an alterate protocol was specified + if (alternateProtocol != null) + { + //Disable transport timeouts + transportContext.ConnectionStream.WriteTimeout = Timeout.Infinite; + transportContext.ConnectionStream.ReadTimeout = Timeout.Infinite; + //Exec the protocol handler and pass the transport stream + await alternateProtocol.RunAsync(transportContext.ConnectionStream, StopToken!.Token); + + //Clear keepalive flag to close the connection + keepalive = false; + } + + return keepalive; + } + finally + { + //Clean end request + context.EndRequest(); + } + } + + /// <summary> + /// Reads data synchronously from the transport and attempts to parse an HTTP message and + /// built a request. + /// </summary> + /// <param name="transport"></param> + /// <param name="ctx"></param> + /// <returns>0 if the request was successfully parsed, the <see cref="HttpStatusCode"/> + /// to return to the client because the entity could not be processed</returns> + /// <remarks> + /// <para> + /// This method is synchronous for multiple memory optimization reasons, + /// and performance is not expected to be reduced as the transport layer should + /// <br></br> + /// only raise an event when a socket has data available to be read, and entity + /// header sections are expected to fit within a single TCP buffer. + /// </para> + /// </remarks> + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + private HttpStatusCode ParseRequest(ITransportContext transport, HttpContext ctx) + { + //Init parser + TransportReader reader = new (transport.ConnectionStream, ctx.RequestBuffer, Config.HttpEncoding, HeaderLineTermination); + + try + { + Span<char> lineBuf = ctx.RequestBuffer.CharBuffer; + + Http11ParseExtensions.Http1ParseState parseState = new(); + + //Parse the request line + HttpStatusCode code = ctx.Request.Http1ParseRequestLine(ref parseState, ref reader, in lineBuf); + + if (code > 0) + { + return code; + } + //Parse the headers + code = ctx.Request.Http1ParseHeaders(ref parseState, ref reader, Config, in lineBuf); + if (code > 0) + { + return code; + } + //Prepare entity body for request + code = ctx.Request.Http1PrepareEntityBody(ref parseState, ref reader, Config); + if (code > 0) + { + return code; + } + //Success! + return 0; + } + //Catch exahusted buffer request + catch (OutOfMemoryException) + { + return HttpStatusCode.RequestHeaderFieldsTooLarge; + } + catch (UriFormatException) + { + return HttpStatusCode.BadRequest; + } + } + + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + private async ValueTask<ERRNO> ProcessRequestAsync(HttpContext context, HttpStatusCode status) + { + //Check status + if (status != 0) + { + /* + * If the status of the parsing was not successfull the transnport is considered + * an unknowns state and could still have data which could corrupt communications + * or worse, contatin an attack. I am choosing to drop the transport and close the + * connection if parsing the request fails + */ + //Close the connection when we exit + context.Response.Headers[HttpResponseHeader.Connection] = "closed"; + //Return status code, if the the expect header was set, return expectation failed, otherwise return the result status code + context.Respond(context.Request.Expect ? HttpStatusCode.ExpectationFailed : status); + //exit and close connection (default result will close the context) + return false; + } + //We only support version 1 and 1/1 + if ((context.Request.HttpVersion & (HttpVersion.Http11 | HttpVersion.Http1)) == 0) + { + //Close the connection when we exit + context.Response.Headers[HttpResponseHeader.Connection] = "closed"; + context.Respond(HttpStatusCode.HttpVersionNotSupported); + return false; + } + //Check open connection count (not super accurate, or might not be atomic) + if (OpenConnectionCount > Config.MaxOpenConnections) + { + //Close the connection and return 503 + context.Response.Headers[HttpResponseHeader.Connection] = "closed"; + context.Respond(HttpStatusCode.ServiceUnavailable); + return false; + } + + //Store keepalive value from request, and check if keepalives are enabled by the configuration + bool keepalive = context.Request.KeepAlive & Config.ConnectionKeepAlive > TimeSpan.Zero; + + //Set connection header (only for http1 and 1.1) + if (keepalive) + { + context.Response.Headers[HttpResponseHeader.Connection] = "keep-alive"; + context.Response.Headers[HttpResponseHeader.KeepAlive] = KeepAliveTimeoutHeaderValue; + } + else + { + //Set connection closed + context.Response.Headers[HttpResponseHeader.Connection] = "closed"; + } + //Get the server root for the specified location + if (!ServerRoots.TryGetValue(context.Request.Location.DnsSafeHost, out IWebRoot? root) && !ServerRoots.TryGetValue(WILDCARD_KEY, out root)) + { + context.Respond(HttpStatusCode.NotFound); + //make sure control leaves + return keepalive; + } + //check for redirects + if (root.Redirects.TryGetValue(context.Request.Location.LocalPath, out Redirect? r)) + { + //301 + context.Redirect301(r.RedirectUrl); + //Return keepalive + return keepalive; + } + //Check the expect header and return an early status code + if (context.Request.Expect) + { + //send a 100 status code + await context.Response.SendEarly100ContinueAsync(); + } + /* + * Initialze the request body state, which may read/buffer the request + * entity body. When doing so, the only exceptions that should be + * generated are IO, OutOfMemory, and Overflow. IOE should + * be raised to the transport as it will only be thrown if the transport + * is in an unusable state. + * + * OOM and Overflow should only be raised if an over-sized entity + * body was allowed to be read in. The Parse method should have guarded + * form data size so oom or overflow would be bugs, and we can let + * them get thrown + */ + await context.Request.InitRequestBodyAsync(Config.FormDataBufferSize, Config.HttpEncoding); + try + { + await ProcessAsync(root, context); + return keepalive; + } + //The user-code requested termination of the connection + catch (TerminateConnectionException tce) + { + //Log the event as a debug so user can see the result + Config.ServerLog.Debug(tce, "User-code requested a connection termination"); + //See if the exception requested an error code response + if (tce.Code > 0) + { + //close response with status code + context.Respond(tce.Code); + } + else + { + //Clear any currently set headers since no response is requested + context.Response.Headers.Clear(); + } + } + return false; + } + + /// <summary> + /// Processes a client connection after pre-processing has completed + /// </summary> + /// <param name="root">The <see cref="IWebRoot"/> to process the event on</param> + /// <param name="ctx">The <see cref="HttpContext"/> to process</param> + /// <returns>A task that resolves when the user-code has completed processing the entity</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="TerminateConnectionException"></exception> + private static async ValueTask ProcessAsync(IWebRoot root, HttpContext ctx) + { + /* + * The event object should be cleared when it is no longer in use, IE before + * this procedure returns. + */ + HttpEvent ev = new (ctx); + try + { + await root.ClientConnectedAsync(ev); + } + //User code requested exit, elevate the exception + catch (TerminateConnectionException) + { + throw; + } + //Transport exception + catch(IOException ioe) when (ioe.InnerException is SocketException) + { + throw; + } + catch (Exception ex) + { + ctx.ParentServer.Config.ServerLog.Warn(ex, "Unhandled exception during application code execution."); + } + finally + { + ev.Clear(); + } + } + + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/IConnectionContext.cs b/Net.Http/src/Core/IConnectionContext.cs new file mode 100644 index 0000000..2e3ca46 --- /dev/null +++ b/Net.Http/src/Core/IConnectionContext.cs @@ -0,0 +1,62 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IConnectionContext.cs +* +* IConnectionContext.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// A request-response stream oriented connection state + /// </summary> + internal interface IConnectionContext + { + /// <summary> + /// Initializes the context to work with the specified + /// transport context + /// </summary> + /// <param name="tranpsort">A referrence to the transport context to use</param> + void InitializeContext(ITransportContext tranpsort); + + /// <summary> + /// Signals the context that it should prepare to process a new request + /// for the current transport + /// </summary> + void BeginRequest(); + + /// <summary> + /// Sends any pending data associated with the request to the + /// connection that begun the request + /// </summary> + /// <param name="cancellationToken">A token to cancel the operation</param> + /// <returns>A Task that completes when the response has completed</returns> + Task WriteResponseAsync(CancellationToken cancellationToken); + + /// <summary> + /// Signals to the context that it will release any request specific + /// resources + /// </summary> + void EndRequest(); + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/IHttpEvent.cs b/Net.Http/src/Core/IHttpEvent.cs new file mode 100644 index 0000000..ec1dbb5 --- /dev/null +++ b/Net.Http/src/Core/IHttpEvent.cs @@ -0,0 +1,104 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IHttpEvent.cs +* +* IHttpEvent.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Collections.Generic; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Contains an http request and session information. + /// </summary> + public interface IHttpEvent + { + /// <summary> + /// Current connection information. (Like "$_SERVER" superglobal in PHP) + /// </summary> + IConnectionInfo Server { get; } + /// <summary> + /// The <see cref="HttpServer"/> that this connection originated from + /// </summary> + HttpServer OriginServer { get; } + + /// <summary> + /// If the request has query arguments they are stored in key value format + /// </summary> + /// <remarks>Keys are case-insensitive</remarks> + IReadOnlyDictionary<string, string> QueryArgs { get; } + /// <summary> + /// If the request body has form data or url encoded arguments they are stored in key value format + /// </summary> + IReadOnlyDictionary<string, string> RequestArgs { get; } + /// <summary> + /// Contains all files upladed with current request + /// </summary> + /// <remarks>Keys are case-insensitive</remarks> + IReadOnlyList<FileUpload> Files { get; } + + /// <summary> + /// Complete the session and respond to user + /// </summary> + /// <param name="code">Status code of operation</param> + /// <exception cref="InvalidOperationException"></exception> + void CloseResponse(HttpStatusCode code); + + /// <summary> + /// Responds to a client with a <see cref="Stream"/> containing data to be sent to user of a given contentType. + /// Runtime will dispose of the stream during closing event + /// </summary> + /// <param name="code">Response status code</param> + /// <param name="type">MIME ContentType of data</param> + /// <param name="stream">Data to be sent to client</param> + /// <exception cref="IOException"></exception> + /// <exception cref="InvalidOperationException"></exception> + void CloseResponse(HttpStatusCode code, ContentType type, Stream stream); + + /// <summary> + /// Responds to a client with an in-memory <see cref="IMemoryResponseReader"/> containing data + /// to be sent to user of a given contentType. + /// </summary> + /// <param name="code">The status code to set</param> + /// <param name="type">The entity content-type</param> + /// <param name="entity">The in-memory response data</param> + /// <exception cref="InvalidOperationException"></exception> + void CloseResponse(HttpStatusCode code, ContentType type, IMemoryResponseReader entity); + + /// <summary> + /// Configures the server to change protocols from HTTP to the specified + /// custom protocol handler. + /// </summary> + /// <param name="protocolHandler">The custom protocol handler</param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="InvalidOperationException"></exception> + void DangerousChangeProtocol(IAlternateProtocol protocolHandler); + + /// <summary> + /// Disables response compression + /// </summary> + void DisableCompression(); + + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/IHttpLifeCycle.cs b/Net.Http/src/Core/IHttpLifeCycle.cs new file mode 100644 index 0000000..135219d --- /dev/null +++ b/Net.Http/src/Core/IHttpLifeCycle.cs @@ -0,0 +1,62 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IHttpLifeCycle.cs +* +* IHttpLifeCycle.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// Represents a interface of lifecycle hooks that correspond + /// with HTTP lifecycle events. + /// </summary> + internal interface IHttpLifeCycle + { + /// <summary> + /// Raised when the context is being prepare for reuse, + /// "revived from storage" + /// </summary> + void OnPrepare(); + + /// <summary> + /// Raised when the context is being released back to the pool + /// for reuse at a later time + /// </summary> + void OnRelease(); + + /// <summary> + /// Raised when a new request is about to be processed + /// on the current context + /// </summary> + void OnNewRequest(); + + /// <summary> + /// Raised when the request has been processed and the + /// response has been sent. Used to perform per-request + /// cleanup/reset for another request. + /// </summary> + /// <remarks> + /// This method is guarunteed to be called regardless of an http error, this + /// method should not throw exceptions + /// </remarks> + void OnComplete(); + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/IHttpResponseBody.cs b/Net.Http/src/Core/IHttpResponseBody.cs new file mode 100644 index 0000000..aa2dd34 --- /dev/null +++ b/Net.Http/src/Core/IHttpResponseBody.cs @@ -0,0 +1,73 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IHttpResponseBody.cs +* +* IHttpResponseBody.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// Represents a rseponse entity body + /// </summary> + internal interface IHttpResponseBody + { + /// <summary> + /// A value that indicates if there is data + /// to send to the client + /// </summary> + bool HasData { get; } + + /// <summary> + /// A value that indicates if response data requires buffering + /// </summary> + bool BufferRequired { get; } + + /// <summary> + /// Writes internal response entity data to the destination stream + /// </summary> + /// <param name="dest">The response stream to write data to</param> + /// <param name="buffer">An optional buffer used to buffer responses</param> + /// <param name="count">The maximum length of the response data to write</param> + /// <param name="token">A token to cancel the operation</param> + /// <returns>A task that resolves when the response is completed</returns> + Task WriteEntityAsync(Stream dest, long count, Memory<byte>? buffer, CancellationToken token); + + /// <summary> + /// Writes internal response entity data to the destination stream + /// </summary> + /// <param name="dest">The response stream to write data to</param> + /// <param name="buffer">An optional buffer used to buffer responses</param> + /// <param name="token">A token to cancel the operation</param> + /// <returns>A task that resolves when the response is completed</returns> + Task WriteEntityAsync(Stream dest, Memory<byte>? buffer, CancellationToken token); + + /// <summary> + /// The length of the content + /// </summary> + long Length { get; } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Request/HttpInputStream.cs b/Net.Http/src/Core/Request/HttpInputStream.cs new file mode 100644 index 0000000..61d215f --- /dev/null +++ b/Net.Http/src/Core/Request/HttpInputStream.cs @@ -0,0 +1,222 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpInputStream.cs +* +* HttpInputStream.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// Specialized stream to allow reading a request entity body with a fixed content length. + /// </summary> + internal sealed class HttpInputStream : Stream + { + private readonly Func<Stream> GetTransport; + + private long ContentLength; + private Stream? InputStream; + private long _position; + + private ISlindingWindowBuffer<byte>? _initalData; + + public HttpInputStream(Func<Stream> getTransport) => GetTransport = getTransport; + + internal void OnComplete() + { + //Dispose the inigial data buffer + _initalData?.Close(); + _initalData = null; + //Remove stream cache copy + InputStream = null; + //Reset position + _position = 0; + //reset content length + ContentLength = 0; + } + + /// <summary> + /// Creates a new input stream object configured to allow reading of the specified content length + /// bytes from the stream and consumes the initial buffer to read data from on initial read calls + /// </summary> + /// <param name="contentLength">The number of bytes to allow being read from the transport or initial buffer</param> + /// <param name="initial">Entity body data captured on initial read</param> + internal void Prepare(long contentLength, ISlindingWindowBuffer<byte>? initial) + { + ContentLength = contentLength; + _initalData = initial; + + //Cache transport + InputStream = GetTransport(); + } + + public override void Close() => throw new NotSupportedException("The HTTP input stream should never be closed!"); + private long Remaining => Math.Max(ContentLength - _position, 0); + public override bool CanRead => true; + public override bool CanSeek => true; + public override bool CanWrite => false; + public override long Length => ContentLength; + public override long Position { get => _position; set { } } + + public override void Flush(){} + + public override int Read(byte[] buffer, int offset, int count) => Read(buffer.AsSpan(offset, count)); + public override int Read(Span<byte> buffer) + { + //Calculate the amount of data that can be read into the buffer + int bytesToRead = (int)Math.Min(buffer.Length, Remaining); + if (bytesToRead == 0) + { + return 0; + } + + //Clamp output buffer size and create buffer writer + ForwardOnlyWriter<byte> writer = new(buffer[..bytesToRead]); + + //See if all data is internally buffered + if (_initalData != null && _initalData.AccumulatedSize > 0) + { + //Read as much as possible from internal buffer + ERRNO read = _initalData.Read(writer.Remaining); + + //Advance writer + writer.Advance(read); + + //Update position + _position += read; + } + + //See if data is still remaining to be read from transport (reamining size is also the amount of data that can be read) + if (writer.RemainingSize > 0) + { + //Read from transport + ERRNO read = InputStream!.Read(writer.Remaining); + + //Update writer position + writer.Advance(read); + + _position += read; + } + + //Return number of bytes written to the buffer + return writer.Written; + } + + public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default) + { + //Calculate the amount of data that can be read into the buffer + int bytesToRead = (int)Math.Min(buffer.Length, Remaining); + if (bytesToRead == 0) + { + return 0; + } + + //Clamp output buffer size and create buffer writer + ForwardOnlyMemoryWriter<byte> writer = new(buffer[..bytesToRead]); + + //See if all data is internally buffered + if (_initalData != null && _initalData.AccumulatedSize > 0) + { + //Read as much as possible from internal buffer + ERRNO read = _initalData.Read(writer.Remaining.Span); + + //Advance writer + writer.Advance(read); + + //Update position + _position += read; + } + + //See if data is still remaining to be read from transport (reamining size is also the amount of data that can be read) + if (writer.RemainingSize > 0) + { + //Read from transport + ERRNO read = await InputStream!.ReadAsync(writer.Remaining, cancellationToken).ConfigureAwait(false); + + //Update writer position + writer.Advance(read); + + _position += read; + } + + //Return number of bytes written to the buffer + return writer.Written; + } + + /// <summary> + /// Asynchronously discards all remaining data in the stream + /// </summary> + /// <param name="heap">The heap to alloc buffers from</param> + /// <param name="maxBufferSize">The maxium size of the buffer to allocate</param> + /// <returns>A task that represents the discard operations</returns> + public async ValueTask DiscardRemainingAsync(int maxBufferSize) + { + long remaining = Remaining; + if(remaining == 0) + { + return; + } + //See if all data has already been buffered + if(_initalData != null && remaining <= _initalData.AccumulatedSize) + { + //All data has been buffred, so just clear the buffer + _position = Length; + } + //We must actaully disacrd data from the stream + else + { + //Calcuate a buffer size to allocate (will never be larger than an int) + int bufferSize = (int)Math.Min(remaining, maxBufferSize); + //Alloc a discard buffer to reset the transport + using IMemoryOwner<byte> discardBuffer = CoreBufferHelpers.GetMemory(bufferSize, false); + int read = 0; + do + { + //Read data to the discard buffer until reading is completed + read = await ReadAsync(discardBuffer.Memory, CancellationToken.None).ConfigureAwait(true); + + } while (read != 0); + } + } + + public override long Seek(long offset, SeekOrigin origin) + { + //Ignore seek control + return _position; + } + public override void SetLength(long value) => throw new NotSupportedException(); + public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Request/HttpRequest.cs b/Net.Http/src/Core/Request/HttpRequest.cs new file mode 100644 index 0000000..2410a8f --- /dev/null +++ b/Net.Http/src/Core/Request/HttpRequest.cs @@ -0,0 +1,284 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpRequest.cs +* +* HttpRequest.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Collections.Generic; +using System.Security.Authentication; + +using VNLib.Utils; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http.Core +{ + + internal class HttpRequest : IHttpLifeCycle +#if DEBUG + ,IStringSerializeable +#endif + { + public readonly VnWebHeaderCollection Headers; + public readonly Dictionary<string, string> Cookies; + public readonly List<string> Accept; + public readonly List<string> AcceptLanguage; + public readonly HttpRequestBody RequestBody; + + public HttpVersion HttpVersion { get; set; } + public HttpMethod Method { get; set; } + public string? UserAgent { get; set; } + public string? Boundry { get; set; } + public ContentType ContentType { get; set; } + public string? Charset { get; set; } + public Uri Location { get; set; } + public Uri? Origin { get; set; } + public Uri? Referrer { get; set; } + internal bool KeepAlive { get; set; } + public IPEndPoint RemoteEndPoint { get; set; } + public IPEndPoint LocalEndPoint { get; set; } + public SslProtocols EncryptionVersion { get; set; } + public Tuple<long, long>? Range { get; set; } + /// <summary> + /// A value indicating whether the connection contained a request entity body. + /// </summary> + public bool HasEntityBody { get; set; } + /// <summary> + /// A transport stream wrapper that is positioned for reading + /// the entity body from the input stream + /// </summary> + public HttpInputStream InputStream { get; } + /// <summary> + /// A value indicating if the client's request had an Expect-100-Continue header + /// </summary> + public bool Expect { get; set; } + +#nullable disable + public HttpRequest(Func<Stream> getTransport) + { + //Create new collection for headers + Headers = new(); + //Create new collection for request cookies + Cookies = new(); + //New list for accept + Accept = new(); + AcceptLanguage = new(); + //New reusable input stream + InputStream = new(getTransport); + RequestBody = new(); + } + + + public void OnPrepare() + {} + + public void OnRelease() + {} + + public void OnNewRequest() + { + //Set to defaults + ContentType = ContentType.NonSupported; + EncryptionVersion = default; + Method = HttpMethod.NOT_SUPPORTED; + HttpVersion = HttpVersion.NotSupported; + } + + public void OnComplete() + { + //release the input stream + InputStream.OnComplete(); + RequestBody.OnComplete(); + //Make sure headers, cookies, and accept are cleared for reuse + Headers.Clear(); + Cookies.Clear(); + Accept.Clear(); + AcceptLanguage.Clear(); + //Clear request flags + this.Expect = false; + this.KeepAlive = false; + this.HasEntityBody = false; + //We need to clean up object refs + this.Boundry = default; + this.Charset = default; + this.LocalEndPoint = default; + this.Location = default; + this.Origin = default; + this.Referrer = default; + this.RemoteEndPoint = default; + this.UserAgent = default; + this.Range = default; + //We are all set to reuse the instance + } + + +#if DEBUG + public string Compile() + { + //Alloc char buffer for compilation + using UnsafeMemoryHandle<char> buffer = Memory.UnsafeAlloc<char>(16 * 1024, true); + ForwardOnlyWriter<char> writer = new(buffer.Span); + Compile(ref writer); + return writer.ToString(); + } + + public void Compile(ref ForwardOnlyWriter<char> writer) + { + //Request line + writer.Append(Method.ToString()); + writer.Append(" "); + writer.Append(Location?.PathAndQuery); + writer.Append(" HTTP/"); + switch (HttpVersion) + { + case HttpVersion.NotSupported: + writer.Append("Unsuppored Http version"); + break; + case HttpVersion.Http1: + writer.Append("1.0"); + break; + case HttpVersion.Http11: + writer.Append("1.1"); + break; + case HttpVersion.Http2: + writer.Append("2.0"); + break; + case HttpVersion.Http09: + writer.Append("0.9"); + break; + } + writer.Append("\r\n"); + //write host + writer.Append("Host: "); + writer.Append(Location?.Authority); + writer.Append("\r\n"); + + //Write headers + foreach (string header in Headers.Keys) + { + writer.Append(header); + writer.Append(": "); + writer.Append(Headers[header]); + writer.Append("\r\n"); + } + //Write cookies + foreach (string cookie in Cookies.Keys) + { + writer.Append("Cookie: "); + writer.Append(cookie); + writer.Append("="); + writer.Append(Cookies[cookie]); + writer.Append("\r\n"); + } + + //Write accept + if (Accept.Count > 0) + { + writer.Append("Accept: "); + foreach (string accept in Accept) + { + writer.Append(accept); + writer.Append(", "); + } + writer.Append("\r\n"); + } + //Write accept language + if (AcceptLanguage.Count > 0) + { + writer.Append("Accept-Language: "); + foreach (string acceptLanguage in AcceptLanguage) + { + writer.Append(acceptLanguage); + writer.Append(", "); + } + writer.Append("\r\n"); + } + //Write user agent + if (UserAgent != null) + { + writer.Append("User-Agent: "); + writer.Append(UserAgent); + writer.Append("\r\n"); + } + //Write content type + if (ContentType != ContentType.NonSupported) + { + writer.Append("Content-Type: "); + writer.Append(HttpHelpers.GetContentTypeString(ContentType)); + writer.Append("\r\n"); + } + //Write content length + if (ContentType != ContentType.NonSupported) + { + writer.Append("Content-Length: "); + writer.Append(InputStream.Length); + writer.Append("\r\n"); + } + if (KeepAlive) + { + writer.Append("Connection: keep-alive\r\n"); + } + if (Expect) + { + writer.Append("Expect: 100-continue\r\n"); + } + if(Origin != null) + { + writer.Append("Origin: "); + writer.Append(Origin.ToString()); + writer.Append("\r\n"); + } + if (Referrer != null) + { + writer.Append("Referrer: "); + writer.Append(Referrer.ToString()); + writer.Append("\r\n"); + } + writer.Append("from "); + writer.Append(RemoteEndPoint.ToString()); + writer.Append("\r\n"); + writer.Append("Received on "); + writer.Append(LocalEndPoint.ToString()); + //Write end of headers + writer.Append("\r\n"); + } + + public ERRNO Compile(in Span<char> buffer) + { + ForwardOnlyWriter<char> writer = new(buffer); + Compile(ref writer); + return writer.Written; + } + public override string ToString() + { + return Compile(); + } +#else + public override string ToString() + { + return "Request debug output only available when compiled in DEBUG mode"; + } +#endif + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Request/HttpRequestBody.cs b/Net.Http/src/Core/Request/HttpRequestBody.cs new file mode 100644 index 0000000..824ca24 --- /dev/null +++ b/Net.Http/src/Core/Request/HttpRequestBody.cs @@ -0,0 +1,70 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpRequestBody.cs +* +* HttpRequestBody.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// Represents a higher-level request entity body (query arguments, request body etc) + /// that has been parsed and captured + /// </summary> + internal class HttpRequestBody + { + public readonly List<FileUpload> Uploads; + public readonly Dictionary<string, string> RequestArgs; + public readonly Dictionary<string, string> QueryArgs; + + public HttpRequestBody() + { + Uploads = new(1); + + //Request/query args should not be request sensitive + RequestArgs = new(StringComparer.OrdinalIgnoreCase); + QueryArgs = new(StringComparer.OrdinalIgnoreCase); + } + + /// <summary> + /// Releases all resources used by the current instance + /// </summary> + public void OnComplete() + { + //Only enumerate/clear if file uplaods are present + if (Uploads.Count > 0) + { + //Dispose all initialized files + for (int i = 0; i < Uploads.Count; i++) + { + Uploads[i].Free(); + } + //Emtpy list + Uploads.Clear(); + } + //Clear request args and file uplaods + RequestArgs.Clear(); + QueryArgs.Clear(); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Request/HttpRequestExtensions.cs b/Net.Http/src/Core/Request/HttpRequestExtensions.cs new file mode 100644 index 0000000..6a93192 --- /dev/null +++ b/Net.Http/src/Core/Request/HttpRequestExtensions.cs @@ -0,0 +1,304 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpRequestExtensions.cs +* +* HttpRequestExtensions.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Text; +using System.Threading.Tasks; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +using static VNLib.Net.Http.Core.CoreBufferHelpers; + +namespace VNLib.Net.Http.Core +{ + internal static class HttpRequestExtensions + { + public enum CompressionType + { + None, + Gzip, + Deflate, + Brotli + } + + /// <summary> + /// Gets the <see cref="CompressionType"/> that the connection accepts + /// in a default order, or none if not enabled + /// </summary> + /// <param name="request"></param> + /// <returns>A <see cref="CompressionType"/> with a value the connection support</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static CompressionType GetCompressionSupport(this HttpRequest request) + { + string? acceptEncoding = request.Headers[HttpRequestHeader.AcceptEncoding]; + + if (acceptEncoding == null) + { + return CompressionType.None; + } + else if (acceptEncoding.Contains("gzip", StringComparison.OrdinalIgnoreCase)) + { + return CompressionType.Gzip; + } + else if (acceptEncoding.Contains("deflate", StringComparison.OrdinalIgnoreCase)) + { + return CompressionType.Deflate; + } + else if (acceptEncoding.Contains("br", StringComparison.OrdinalIgnoreCase)) + { + return CompressionType.Brotli; + } + else + { + return CompressionType.None; + } + } + + + /// <summary> + /// Tests the connection's origin header against the location URL by authority. + /// An origin matches if its scheme, host, and port match + /// </summary> + /// <returns>true if the origin header was set and does not match the current locations origin</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsCrossOrigin(this HttpRequest Request) + { + return Request.Origin != null + && (!Request.Origin.Authority.Equals(Request.Location.Authority, StringComparison.Ordinal) + || !Request.Origin.Scheme.Equals(Request.Location.Scheme, StringComparison.Ordinal)); + } + /// <summary> + /// Is the current connection a websocket upgrade request handshake + /// </summary> + /// <returns>true if the connection is a websocket upgrade request, false otherwise</returns> + public static bool IsWebSocketRequest(this HttpRequest Request) + { + string? upgrade = Request.Headers[HttpRequestHeader.Upgrade]; + if (!string.IsNullOrWhiteSpace(upgrade) && upgrade.Contains("websocket", StringComparison.OrdinalIgnoreCase)) + { + //This request is a websocket request + //Check connection header + string? connection = Request.Headers[HttpRequestHeader.Connection]; + //Must be a web socket request + return !string.IsNullOrWhiteSpace(connection) && connection.Contains("upgrade", StringComparison.OrdinalIgnoreCase); + } + return false; + } + + /// <summary> + /// Initializes the <see cref="HttpRequest"/> for an incomming connection + /// </summary> + /// <param name="server"></param> + /// <param name="ctx">The <see cref="TransportEventContext"/> to attach the request to</param> + /// <param name="defaultHttpVersion">The default http version</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Initialize(this HttpRequest server, ITransportContext ctx, HttpVersion defaultHttpVersion) + { + server.LocalEndPoint = ctx.LocalEndPoint; + server.RemoteEndPoint = ctx.RemoteEndpoint; + server.EncryptionVersion = ctx.SslVersion; + //Set to default http version so the response can be configured properly + server.HttpVersion = defaultHttpVersion; + } + + + /// <summary> + /// Initializes the <see cref="HttpRequest.RequestBody"/> for the current request + /// </summary> + /// <param name="Request"></param> + /// <param name="maxBufferSize">The maxium buffer size allowed while parsing reqeust body data</param> + /// <param name="encoding">The request data encoding for url encoded or form data bodies</param> + /// <exception cref="IOException"></exception> + /// <exception cref="OverflowException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + internal static ValueTask InitRequestBodyAsync(this HttpRequest Request, int maxBufferSize, Encoding encoding) + { + /* + * Parses query parameters from the request location query + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static void ParseQueryArgs(HttpRequest Request) + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + //Query string parse method + static void QueryParser(ReadOnlySpan<char> queryArgument, HttpRequest Request) + { + //Split spans after the '=' character + ReadOnlySpan<char> key = queryArgument.SliceBeforeParam('='); + ReadOnlySpan<char> value = queryArgument.SliceAfterParam('='); + //Insert into dict + Request.RequestBody.QueryArgs[key.ToString()] = value.ToString(); + } + + //if the request has query args, parse and store them + ReadOnlySpan<char> queryString = Request.Location.Query; + if (!queryString.IsEmpty) + { + //trim leading '?' if set + queryString = queryString.TrimStart('?'); + //Split args by '&' + queryString.Split('&', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries, QueryParser, Request); + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static async ValueTask ParseInputStream(HttpRequest Request, int maxBufferSize, Encoding encoding) + { + /* + * Reads all available data from the request input stream + * If the stream size is smaller than a TCP buffer size, will complete synchronously + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static ValueTask<VnString> ReadInputStreamAsync(HttpRequest Request, int maxBufferSize, Encoding encoding) + { + //Calculate a largest available buffer to read the entire stream or up to the maximum buffer size + int bufferSize = (int)Math.Min(Request.InputStream.Length, maxBufferSize); + //Read the stream into a vnstring + return VnString.FromStreamAsync(Request.InputStream, encoding, HttpPrivateHeap, bufferSize); + } + /* + * SpanSplit callback function for storing UrlEncoded request entity + * bodies in key-value pairs and writing them to the + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static void UrlEncodedSplitCb(ReadOnlySpan<char> kvArg, HttpRequest Request) + { + //Get key side of agument (or entire argument if no value is set) + ReadOnlySpan<char> key = kvArg.SliceBeforeParam('='); + ReadOnlySpan<char> value = kvArg.SliceAfterParam('='); + //trim, allocate strings, and store in the request arg dict + Request.RequestBody.RequestArgs[key.TrimCRLF().ToString()] = value.TrimCRLF().ToString(); + } + + /* + * Parses a Form-Data content type request entity body and stores those arguments in + * Request uploads or request args + */ + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static void FormDataBodySplitCb(ReadOnlySpan<char> formSegment, ValueTuple<HttpRequestBody, Encoding> state) + { + //Form data arguments + string? DispType = null, Name = null, FileName = null; + ContentType ctHeaderVal = ContentType.NonSupported; + //Get sliding window for parsing data + ForwardOnlyReader<char> reader = new(formSegment.TrimCRLF()); + //Read content headers + do + { + //Get the index of the next crlf + int index = reader.Window.IndexOf(HttpHelpers.CRLF); + //end of headers + if (index < 1) + { + break; + } + //Get header data + ReadOnlySpan<char> header = reader.Window[..index]; + //Split header at colon + int colon = header.IndexOf(':'); + //If no data is available after the colon the header is not valid, so move on to the next body + if (colon < 1) + { + return; + } + //Hash the header value into a header enum + HttpRequestHeader headerType = HttpHelpers.GetRequestHeaderEnumFromValue(header[..colon]); + //get the header value + ReadOnlySpan<char> headerValue = header[(colon + 1)..]; + //Check for content dispositon header + if (headerType == HttpHelpers.ContentDisposition) + { + //Parse the content dispostion + HttpHelpers.ParseDisposition(headerValue, out DispType, out Name, out FileName); + } + //Check for content type + else if (headerType == HttpRequestHeader.ContentType) + { + //The header value for content type should be an MIME content type + ctHeaderVal = HttpHelpers.GetContentType(headerValue.Trim().ToString()); + } + //Shift window to the next line + reader.Advance(index + HttpHelpers.CRLF.Length); + } while (true); + //Remaining data should be the body data (will have leading and trailing CRLF characters + //If filename is set, this must be a file + if (!string.IsNullOrWhiteSpace(FileName)) + { + //Store the file in the uploads + state.Item1.Uploads.Add(FileUpload.FromString(reader.Window.TrimCRLF(), state.Item2, FileName, ctHeaderVal)); + } + //Make sure the name parameter was set and store the message body as a string + else if (!string.IsNullOrWhiteSpace(Name)) + { + //String data as body + state.Item1.RequestArgs[Name] = reader.Window.TrimCRLF().ToString(); + } + } + + switch (Request.ContentType) + { + //CT not supported, dont read it + case ContentType.NonSupported: + break; + case ContentType.UrlEncoded: + //Create a vnstring from the message body and parse it (assuming url encoded bodies are small so a small stack buffer will be fine) + using (VnString urlbody = await ReadInputStreamAsync(Request, maxBufferSize, encoding)) + { + //Get the body as a span, and split the 'string' at the & character + urlbody.AsSpan().Split('&', StringSplitOptions.RemoveEmptyEntries, UrlEncodedSplitCb, Request); + } + break; + case ContentType.MultiPart: + //Make sure we have a boundry specified + if (string.IsNullOrWhiteSpace(Request.Boundry)) + { + break; + } + //Read all data from stream into string + using (VnString body = await ReadInputStreamAsync(Request, maxBufferSize, encoding)) + { + //Split the body as a span at the boundries + body.AsSpan().Split($"--{Request.Boundry}", StringSplitOptions.RemoveEmptyEntries, FormDataBodySplitCb, (Request.RequestBody, encoding)); + } + break; + //Default case is store as a file + default: + //add upload + Request.RequestBody.Uploads.Add(new(Request.InputStream, string.Empty, Request.ContentType, false)); + break; + } + } + + //Parse query + ParseQueryArgs(Request); + + //Decode requests from body + return !Request.HasEntityBody ? ValueTask.CompletedTask : ParseInputStream(Request, maxBufferSize, encoding); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/RequestParse/Http11ParseExtensions.cs b/Net.Http/src/Core/RequestParse/Http11ParseExtensions.cs new file mode 100644 index 0000000..b37b78b --- /dev/null +++ b/Net.Http/src/Core/RequestParse/Http11ParseExtensions.cs @@ -0,0 +1,533 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: Http11ParseExtensions.cs +* +* Http11ParseExtensions.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Linq; +using System.Collections.Generic; +using System.Security.Authentication; +using System.Runtime.CompilerServices; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Logging; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http.Core +{ + + internal static class Http11ParseExtensions + { + + /// <summary> + /// Stores the state of an HTTP/1.1 parsing operation + /// </summary> + public ref struct Http1ParseState + { + internal UriBuilder? Location; + internal bool IsAbsoluteRequestUrl; + internal long ContentLength; + } + + + /// <summary> + /// Reads the first line from the transport stream using the specified buffer + /// and parses the HTTP request line components: Method, resource, Http Version + /// </summary> + /// <param name="Request"></param> + /// <param name="reader">The reader to read lines from the transport</param> + /// <param name="parseState">The HTTP1 parsing state</param> + /// <param name="lineBuf">The buffer to use when parsing the request data</param> + /// <returns>0 if the request line was successfully parsed, a status code if the request could not be processed</returns> + /// <exception cref="UriFormatException"></exception> + [MethodImpl(MethodImplOptions.AggressiveOptimization | MethodImplOptions.AggressiveInlining)] + public static HttpStatusCode Http1ParseRequestLine(this HttpRequest Request, ref Http1ParseState parseState, ref TransportReader reader, in Span<char> lineBuf) + { + //Locals + ERRNO requestResult; + int index, endloc; + + //Read the start line + requestResult = reader.ReadLine(lineBuf); + //Must be able to parse the verb and location + if (requestResult < 1) + { + //empty request + return (HttpStatusCode)1000; + } + + //true up the request line to actual size + ReadOnlySpan<char> requestLine = lineBuf[..(int)requestResult].Trim(); + //Find the first white space character ("GET / HTTP/1.1") + index = requestLine.IndexOf(' '); + if (index == -1) + { + return HttpStatusCode.BadRequest; + } + + //Decode the verb (function requires the string be the exact characters of the request method) + Request.Method = HttpHelpers.GetRequestMethod(requestLine[0..index]); + //Make sure the method is supported + if (Request.Method == HttpMethod.NOT_SUPPORTED) + { + return HttpStatusCode.MethodNotAllowed; + } + + //location string should be from end of verb to HTTP/ NOTE: Only supports http... this is an http server + endloc = requestLine.LastIndexOf(" HTTP/", StringComparison.OrdinalIgnoreCase); + //Client must specify an http version prepended by a single whitespace(rfc2612) + if (endloc == -1) + { + return HttpStatusCode.HttpVersionNotSupported; + } + + //Try to parse the version and only accept the 3 major versions of http + Request.HttpVersion = HttpHelpers.ParseHttpVersion(requestLine[endloc..]); + //Check to see if the version was parsed succesfully + if (Request.HttpVersion == HttpVersion.NotSupported) + { + //Return not supported + return HttpStatusCode.HttpVersionNotSupported; + } + + //Set keepalive flag if http11 + Request.KeepAlive = Request.HttpVersion == HttpVersion.Http11; + + //Get the location segment from the request line + ReadOnlySpan<char> paq = requestLine[(index + 1)..endloc].TrimCRLF(); + + //Process an absolute uri, + if (paq.Contains("://", StringComparison.Ordinal)) + { + //Convert the location string to a .net string and init the location builder (will perform validation when the Uri propery is used) + parseState.Location = new(paq.ToString()); + parseState.IsAbsoluteRequestUrl = true; + return 0; + } + //Try to capture a realative uri + else if (paq.Length > 0 && paq[0] == '/') + { + //Create a default location uribuilder + parseState.Location = new() + { + //Set a default scheme + Scheme = Request.EncryptionVersion == SslProtocols.None ? Uri.UriSchemeHttp : Uri.UriSchemeHttps, + }; + //Need to manually parse the query string + int q = paq.IndexOf('?'); + //has query? + if (q == -1) + { + parseState.Location.Path = paq.ToString(); + } + //Does have query argument + else + { + //separate the path from the query + parseState.Location.Path = paq[0..q].ToString(); + parseState.Location.Query = paq[(q + 1)..].ToString(); + } + return 0; + } + //Cannot service an unknonw location + return HttpStatusCode.BadRequest; + } + + /// <summary> + /// Reads headers from the transport using the supplied character buffer, and updates the current request + /// </summary> + /// <param name="Request"></param> + /// <param name="parseState">The HTTP1 parsing state</param> + /// <param name="Config">The current server <see cref="HttpConfig"/></param> + /// <param name="reader">The <see cref="VnStreamReader"/> to read lines from the transport</param> + /// <param name="lineBuf">The buffer read data from the transport with</param> + /// <returns>0 if the request line was successfully parsed, a status code if the request could not be processed</returns> + [MethodImpl(MethodImplOptions.AggressiveOptimization | MethodImplOptions.AggressiveInlining)] + public static HttpStatusCode Http1ParseHeaders(this HttpRequest Request, ref Http1ParseState parseState, ref TransportReader reader, in HttpConfig Config, in Span<char> lineBuf) + { + try + { + int headerCount = 0, colon; + bool hostFound = false; + ERRNO charsRead; + ReadOnlySpan<char> headerName, requestHeaderValue; + + /* + * This loop will read "lines" from the transport/reader buffer as headers + * and store them in the rented character buffer with 0 allocations. + * + * Lines will be read from the transport reader until an empty line is read, + * or an exception occurs. The VnStreamReader class will search for lines + * directly in the binary rather than converting the data then parsing it. + * When a line is parsed, its assumed to be an HTTP header at this point in + * the parsing, and is separated into its key-value pair determined by the + * first ':' character to appear. + * + * The header length will be limited by the size of the character buffer, + * or the reader binary buffer while reading lines. Buffer sizes are fixed + * to the system memory page size. Depending on the encoding the user chooses + * this should not be an issue for most configurations. This strategy is + * most efficient for realtivly small header sizes. + * + * The header's key is hashed by the HttpHelpers class and the hash is used to + * index a lookup table to return its enumeration value which is used in the swtich + * statement to reduce the number of strings added to the request header container. + * This was a major effort to reduce memory and CPU overhead while using the + * WebHeaderCollection .NET class, which I think is still worth using instead of a + * custom header data structure class. + * + * Some case statments are custom HttpRequestHeader enum values via internal casted + * constants to be consistant with he .NET implementation. + */ + do + { + //Read a line until we reach the end of headers, this call will block if end of characters is reached and a new string will be read + charsRead = reader.ReadLine(lineBuf); + + //If the result is less than 1, no line is available (end of headers) or could not be read + if (charsRead < 1) + { + break; + } + + //Header count exceeded or header larger than header buffer size + if (charsRead < 0 || headerCount > Config.MaxRequestHeaderCount) + { + return HttpStatusCode.RequestHeaderFieldsTooLarge; + } + + { + //Get the true size of the read header line as a readonly span + ReadOnlySpan<char> header = lineBuf[..(int)charsRead]; + + /* + * RFC 7230, ignore headers with preceeding whitespace + * + * If the first character is whitespace that is enough to + * ignore the rest of the header + */ + if (header[0] == ' ') + { + //Move on to next header + continue; + } + + //Find the first colon + colon = header.IndexOf(':'); + //No colon was found, this is an invalid string, try to skip it and keep reading + if (colon <= 0) + { + continue; + } + + //Store header and its value (sections before and after colon) + headerName = header[..colon].TrimCRLF(); + requestHeaderValue = header[(colon + 1)..].TrimCRLF(); + } + + //Hash the header key and lookup the request header value + switch (HttpHelpers.GetRequestHeaderEnumFromValue(headerName)) + { + case HttpRequestHeader.Connection: + { + //Update keepalive, if the connection header contains "closed" and with the current value of keepalive + Request.KeepAlive &= !requestHeaderValue.Contains("close", StringComparison.OrdinalIgnoreCase); + //Also store the connecion header into the store + Request.Headers.Add(HttpRequestHeader.Connection, requestHeaderValue.ToString()); + } + break; + case HttpRequestHeader.ContentType: + { + if (!HttpHelpers.TryParseContentType(requestHeaderValue.ToString(), out string? ct, out string? charset, out string? boundry) || ct == null) + { + //Invalid content type header value + return HttpStatusCode.UnsupportedMediaType; + } + Request.Boundry = boundry; + Request.Charset = charset; + //Get the content type enum from mime type + Request.ContentType = HttpHelpers.GetContentType(ct); + } + break; + case HttpRequestHeader.ContentLength: + { + //Content length has already been calculated, ERROR, rfc 7230 + if(parseState.ContentLength > 0) + { + Config.ServerLog.Debug("Message warning, recieved multiple content length headers"); + return HttpStatusCode.BadRequest; + } + + //Only capture positive values, and if length is negative we are supposed to ignore it + if (ulong.TryParse(requestHeaderValue, out ulong len) && len < long.MaxValue) + { + parseState.ContentLength = (long)len; + } + else + { + return HttpStatusCode.BadRequest; + } + + //Request size it too large to service + if (parseState.ContentLength > Config.MaxUploadSize) + { + return HttpStatusCode.RequestEntityTooLarge; + } + } + break; + case HttpRequestHeader.Host: + { + //Set host found flag + hostFound = true; + + //Split the host value by the port parameter + ReadOnlySpan<char> port = requestHeaderValue.SliceAfterParam(':').Trim(); + //Slicing beofre the colon should always provide a useable hostname, so allocate a string for it + string host = requestHeaderValue.SliceBeforeParam(':').Trim().ToString(); + + //Verify that the host is usable + if (Uri.CheckHostName(host) == UriHostNameType.Unknown) + { + return HttpStatusCode.BadRequest; + } + + //Verify that the host matches the host header if absolue uri is set + if (parseState.IsAbsoluteRequestUrl) + { + if (!host.Equals(parseState.Location!.Host, StringComparison.OrdinalIgnoreCase)) + { + return HttpStatusCode.BadRequest; + } + } + + //store the host value + parseState.Location!.Host = host; + + //If the port span is empty, no colon was found or the port is invalid + if (!port.IsEmpty) + { + //try to parse the port number + if (!int.TryParse(port, out int p) || p < 0 || p > ushort.MaxValue) + { + return HttpStatusCode.BadRequest; + } + //Store port + parseState.Location.Port = p; + } + } + break; + case HttpRequestHeader.Cookie: + { + //Local function to break cookie segments into key-value pairs + static void AddCookiesCallback(ReadOnlySpan<char> cookie, Dictionary<string, string> cookieContainer) + { + //Get the name parameter and alloc a string + string name = cookie.SliceBeforeParam('=').Trim().ToString(); + //Get the value parameter and alloc a string + string value = cookie.SliceAfterParam('=').Trim().ToString(); + //Add the cookie to the dictionary + _ = cookieContainer.TryAdd(name, value); + } + //Split all cookies by ; with trailing whitespace + requestHeaderValue.Split("; ", StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries, AddCookiesCallback, Request.Cookies); + } + break; + case HttpRequestHeader.AcceptLanguage: + //Capture accept languages and store in the request accept collection + requestHeaderValue.Split(',', Request.AcceptLanguage, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + break; + case HttpRequestHeader.Accept: + //Capture accept content types and store in request accept collection + requestHeaderValue.Split(',', Request.Accept, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + break; + case HttpRequestHeader.Referer: + { + //Check the referer header and capture its uri instance, it should be absolutely parseable + if (!requestHeaderValue.IsEmpty && Uri.TryCreate(requestHeaderValue.ToString(), UriKind.Absolute, out Uri? refer)) + { + Request.Referrer = refer; + } + } + break; + case HttpRequestHeader.Range: + { + //See if range bytes value has been set + ReadOnlySpan<char> rawRange = requestHeaderValue.SliceAfterParam("bytes=").TrimCRLF(); + //Make sure the bytes parameter is set + if (rawRange.IsEmpty) + { + break; + } + //Get start range + ReadOnlySpan<char> startRange = rawRange.SliceBeforeParam('-'); + //Get end range (empty if no - exists) + ReadOnlySpan<char> endRange = rawRange.SliceAfterParam('-'); + //See if a range end is specified + if (endRange.IsEmpty) + { + //No end range specified, so only range start + if (long.TryParse(startRange, out long start) && start > -1) + { + //Create new range + Request.Range = new(start, -1); + break; + } + } + //Range has a start and end + else if (long.TryParse(startRange, out long start) && long.TryParse(endRange, out long end) && end > -1) + { + //get start and end components from range header + Request.Range = new(start, end); + break; + } + } + //Could not parse start range from header + return HttpStatusCode.RequestedRangeNotSatisfiable; + case HttpRequestHeader.UserAgent: + //Store user-agent + Request.UserAgent = requestHeaderValue.IsEmpty ? string.Empty : requestHeaderValue.TrimCRLF().ToString(); + break; + //Special code for origin header + case HttpHelpers.Origin: + { + //Alloc a string for origin + string origin = requestHeaderValue.ToString(); + //Origin headers should always be absolute address "parsable" + if (Uri.TryCreate(origin, UriKind.Absolute, out Uri? org)) + { + Request.Origin = org; + } + } + break; + case HttpRequestHeader.Expect: + //Accept 100-continue for the Expect header value + Request.Expect = requestHeaderValue.Equals("100-continue", StringComparison.OrdinalIgnoreCase); + break; + default: + //By default store the header in the request header store + Request.Headers.Add(headerName.ToString(), requestHeaderValue.ToString()); + break; + } + //Increment header count + headerCount++; + } while (true); + + //If request is http11 then host is required + if (Request.HttpVersion == HttpVersion.Http11 && !hostFound) + { + return HttpStatusCode.BadRequest; + } + + } + //Catch an arugment exception within the header add function to cause a bad request result + catch (ArgumentException) + { + return HttpStatusCode.BadRequest; + } + + //Check the final location to make sure data was properly sent + if (string.IsNullOrWhiteSpace(parseState.Location?.Host) + || string.IsNullOrWhiteSpace(parseState.Location.Scheme) + || string.IsNullOrWhiteSpace(parseState.Location.Path) + ) + { + return HttpStatusCode.BadRequest; + } + + //Store the finalized location + Request.Location = parseState.Location.Uri; + + return 0; + } + /// <summary> + /// Prepares the entity body for the current HTTP1 request + /// </summary> + /// <param name="Request"></param> + /// <param name="Config">The current server <see cref="HttpConfig"/></param> + /// <param name="parseState">The HTTP1 parsing state</param> + /// <param name="reader">The <see cref="VnStreamReader"/> to read lines from the transport</param> + /// <returns>0 if the request line was successfully parsed, a status code if the request could not be processed</returns> + [MethodImpl(MethodImplOptions.AggressiveOptimization | MethodImplOptions.AggressiveInlining)] + public static HttpStatusCode Http1PrepareEntityBody(this HttpRequest Request, ref Http1ParseState parseState, ref TransportReader reader, in HttpConfig Config) + { + //If the content type is multipart, make sure its not too large to ingest + if (Request.ContentType == ContentType.MultiPart && parseState.ContentLength > Config.MaxFormDataUploadSize) + { + return HttpStatusCode.RequestEntityTooLarge; + } + + //Only ingest the rest of the message body if the request is not a head, get, or trace methods + if ((Request.Method & (HttpMethod.GET | HttpMethod.HEAD | HttpMethod.TRACE)) != 0) + { + //Bad format to include a message body with a GET, HEAD, or TRACE request + if (parseState.ContentLength > 0) + { + Config.ServerLog.Debug("Message body received from {ip} with GET, HEAD, or TRACE request, was considered an error and the request was dropped", Request.RemoteEndPoint); + return HttpStatusCode.BadRequest; + } + else + { + //Success! + return 0; + } + } + + //Check for chuncked transfer encoding + ReadOnlySpan<char> transfer = Request.Headers[HttpRequestHeader.TransferEncoding]; + if (!transfer.IsEmpty && transfer.Contains("chunked", StringComparison.OrdinalIgnoreCase)) + { + //Not a valid http version for chunked transfer encoding + if (Request.HttpVersion != HttpVersion.Http11) + { + return HttpStatusCode.BadRequest; + } + /* + * Was a content length also specified? + * This is an issue and is likely an attack. I am choosing not to support + * the HTTP 1.1 standard and will deny reading the rest of the data from the + * transport. + */ + if (parseState.ContentLength > 0) + { + Config.ServerLog.Debug("Possible attempted desync, Content length + chunked encoding specified. RemoteEP: {ip}", Request.RemoteEndPoint); + return HttpStatusCode.BadRequest; + } + + //Handle chunked transfer encoding (not implemented yet) + return HttpStatusCode.NotImplemented; + } + //Make sure non-zero cl header was provided + else if (parseState.ContentLength > 0) + { + //Open a temp buffer to store initial data in + ISlindingWindowBuffer<byte>? initData = reader.GetReminaingData(parseState.ContentLength); + //Setup the input stream and capture the initial data from the reader, and wrap the transport stream to read data directly + Request.InputStream.Prepare(parseState.ContentLength, initData); + Request.HasEntityBody = true; + } + //Success! + return 0; + } + } +} diff --git a/Net.Http/src/Core/Response/ChunkDataAccumulator.cs b/Net.Http/src/Core/Response/ChunkDataAccumulator.cs new file mode 100644 index 0000000..35c0275 --- /dev/null +++ b/Net.Http/src/Core/Response/ChunkDataAccumulator.cs @@ -0,0 +1,228 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ChunkDataAccumulator.cs +* +* ChunkDataAccumulator.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils; +using VNLib.Utils.IO; + +using static VNLib.Net.Http.Core.CoreBufferHelpers; + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// A specialized <see cref="IDataAccumulator{T}"/> for buffering data + /// in Http/1.1 chunks + /// </summary> + internal class ChunkDataAccumulator : IDataAccumulator<byte>, IHttpLifeCycle + { + public const int RESERVED_CHUNK_SUGGESTION = 32; + + private readonly int BufferSize; + private readonly int ReservedSize; + private readonly Encoding Encoding; + private readonly ReadOnlyMemory<byte> CRLFBytes; + + public ChunkDataAccumulator(Encoding encoding, int bufferSize) + { + Encoding = encoding; + CRLFBytes = encoding.GetBytes(HttpHelpers.CRLF); + + ReservedSize = RESERVED_CHUNK_SUGGESTION; + BufferSize = bufferSize; + } + + private byte[]? _buffer; + private int _reservedOffset; + + + ///<inheritdoc/> + public int RemainingSize => _buffer!.Length - AccumulatedSize; + ///<inheritdoc/> + public Span<byte> Remaining => _buffer!.AsSpan(AccumulatedSize); + ///<inheritdoc/> + public Span<byte> Accumulated => _buffer!.AsSpan(_reservedOffset, AccumulatedSize); + ///<inheritdoc/> + public int AccumulatedSize { get; set; } + + private Memory<byte> CompleteChunk => _buffer.AsMemory(_reservedOffset, (AccumulatedSize - _reservedOffset)); + + /// <summary> + /// Attempts to buffer as much data as possible from the specified data + /// </summary> + /// <param name="data">The data to copy</param> + /// <returns>The number of bytes that were buffered</returns> + public ERRNO TryBufferChunk(ReadOnlySpan<byte> data) + { + //Calc data size and reserve space for final crlf + int dataToCopy = Math.Min(data.Length, RemainingSize - CRLFBytes.Length); + + //Write as much data as possible + data[..dataToCopy].CopyTo(Remaining); + //Advance buffer + Advance(dataToCopy); + + //Return number of bytes not written + return dataToCopy; + } + + ///<inheritdoc/> + public void Advance(int count) => AccumulatedSize += count; + + private void InitReserved() + { + //First reserve the chunk window by advancing the accumulator to the size + Advance(ReservedSize); + } + + ///<inheritdoc/> + public void Reset() + { + //zero offsets + _reservedOffset = 0; + AccumulatedSize = 0; + //Init reserved segment + InitReserved(); + } + + /// <summary> + /// Writes the buffered data as a single chunk to the stream asynchronously. The internal + /// state is reset if writing compleded successfully + /// </summary> + /// <param name="output">The stream to write data to</param> + /// <param name="cancellation">A token to cancel the operation</param> + /// <returns>A value task that resolves when the data has been written to the stream</returns> + public async ValueTask FlushAsync(Stream output, CancellationToken cancellation) + { + //Update the chunk size + UpdateChunkSize(); + + //Write trailing chunk delimiter + this.Append(CRLFBytes.Span); + + //write to stream + await output.WriteAsync(CompleteChunk, cancellation); + + //Reset for next chunk + Reset(); + } + + /// <summary> + /// Writes the buffered data as a single chunk to the stream. The internal + /// state is reset if writing compleded successfully + /// </summary> + /// <param name="output">The stream to write data to</param> + /// <returns>A value task that resolves when the data has been written to the stream</returns> + public void Flush(Stream output) + { + //Update the chunk size + UpdateChunkSize(); + + //Write trailing chunk delimiter + this.Append(CRLFBytes.Span); + + //write to stream + output.Write(CompleteChunk.Span); + + //Reset for next chunk + Reset(); + } + + private void UpdateChunkSize() + { + const int CharBufSize = 2 * sizeof(int); + + /* + * Alloc stack buffer to store chunk size hex chars + * the size of the buffer should be at least the number + * of bytes of the max chunk size + */ + Span<char> s = stackalloc char[CharBufSize]; + + //Chunk size is the accumulated size without the reserved segment + int chunkSize = (AccumulatedSize - ReservedSize); + + //format the chunk size + chunkSize.TryFormat(s, out int written, "x"); + + //temp buffer to store encoded data in + Span<byte> encBuf = stackalloc byte[ReservedSize]; + //Encode the chunk size chars + int initOffset = Encoding.GetBytes(s[..written], encBuf); + + Span<byte> encoded = encBuf[..initOffset]; + + /* + * We need to calcuate how to store the encoded buffer directly + * before the accumulated chunk data. + * + * This requires us to properly upshift the reserved buffer to + * the exact size required to store the encoded chunk size + */ + + _reservedOffset = (ReservedSize - (initOffset + CRLFBytes.Length)); + + Span<byte> upshifted = _buffer!.AsSpan(_reservedOffset, ReservedSize); + + //First write the chunk size + encoded.CopyTo(upshifted); + + //Upshift again to write the crlf + upshifted = upshifted[initOffset..]; + + //Copy crlf + CRLFBytes.Span.CopyTo(upshifted); + } + + + public void OnNewRequest() + { + InitReserved(); + } + + public void OnComplete() + { + //Zero offsets + _reservedOffset = 0; + AccumulatedSize = 0; + } + + public void OnPrepare() + { + //Alloc buffer + _buffer = HttpBinBufferPool.Rent(BufferSize); + } + + public void OnRelease() + { + HttpBinBufferPool.Return(_buffer!); + _buffer = null; + } + + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Response/ChunkedStream.cs b/Net.Http/src/Core/Response/ChunkedStream.cs new file mode 100644 index 0000000..7a8bebc --- /dev/null +++ b/Net.Http/src/Core/Response/ChunkedStream.cs @@ -0,0 +1,249 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ChunkedStream.cs +* +* ChunkedStream.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +/* +* Provides a Chunked data-encoding stream for writing data-chunks to +* the transport using the basic chunked encoding format from MDN +* https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding#directives +* +* This stream will buffer entire chunks to avoid multiple writes to the +* transport which can block or at minium cause overhead in context switching +* which should be mostly avoided but cause overhead in copying. Time profiling +* showed nearly equivalent performance for small chunks for synchronous writes. +* +*/ + +using System; +using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils; +using VNLib.Utils.Memory; + +namespace VNLib.Net.Http.Core +{ + + internal partial class HttpResponse + { + /// <summary> + /// Writes chunked HTTP message bodies to an underlying streamwriter + /// </summary> + private class ChunkedStream : Stream, IHttpLifeCycle + { + private const string LAST_CHUNK_STRING = "0\r\n\r\n"; + + private readonly ReadOnlyMemory<byte> LastChunk; + private readonly ChunkDataAccumulator ChunckAccumulator; + private readonly Func<Stream> GetTransport; + + private Stream? TransportStream; + private bool HadError; + + internal ChunkedStream(Encoding encoding, int chunkBufferSize, Func<Stream> getStream) + { + //Convert and store cached versions of the last chunk bytes + LastChunk = encoding.GetBytes(LAST_CHUNK_STRING); + + //get the min buffer by rounding to the nearest page + int actualBufSize = (chunkBufferSize / 4096 + 1) * 4096; + + //Init accumulator + ChunckAccumulator = new(encoding, actualBufSize); + + GetTransport = getStream; + } + + + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + public override int Read(byte[] buffer, int offset, int count) => throw new NotSupportedException("This stream cannot be read from"); + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException("This stream does not support seeking"); + public override void SetLength(long value) => throw new NotSupportedException("This stream does not support seeking"); + public override void Flush() { } + public override Task FlushAsync(CancellationToken cancellationToken) => Task.CompletedTask; + + + public override void Write(byte[] buffer, int offset, int count) => Write(new ReadOnlySpan<byte>(buffer, offset, count)); + public override void Write(ReadOnlySpan<byte> chunk) + { + //Only write non-zero chunks + if (chunk.Length <= 0) + { + return; + } + + //Init reader + ForwardOnlyReader<byte> reader = new(in chunk); + try + { + do + { + //try to accumulate the chunk data + ERRNO written = ChunckAccumulator.TryBufferChunk(reader.Window); + + //Not all data was buffered + if (written < reader.WindowSize) + { + //Advance reader + reader.Advance(written); + + //Flush accumulator + ChunckAccumulator.Flush(TransportStream!); + //Continue to buffer / flush as needed + continue; + } + break; + } + while (true); + } + catch + { + HadError = true; + throw; + } + } + + + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + return WriteAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + + public override async ValueTask WriteAsync(ReadOnlyMemory<byte> chunk, CancellationToken cancellationToken = default) + { + //Only write non-zero chunks + if (chunk.Length <= 0) + { + return; + } + + try + { + //Init reader + ForwardOnlyMemoryReader<byte> reader = new(in chunk); + + do + { + //try to accumulate the chunk data + ERRNO written = ChunckAccumulator.TryBufferChunk(reader.Window.Span); + + //Not all data was buffered + if (written < reader.WindowSize) + { + //Advance reader + reader.Advance(written); + + //Flush accumulator async + await ChunckAccumulator.FlushAsync(TransportStream!, cancellationToken); + //Continue to buffer / flush as needed + continue; + } + break; + } + while (true); + } + catch + { + HadError = true; + throw; + } + } + + public override async ValueTask DisposeAsync() + { + //If write error occured, then do not write the last chunk + if (HadError) + { + return; + } + + //Write remaining data to stream + await ChunckAccumulator.FlushAsync(TransportStream!, CancellationToken.None); + + //Write final chunk + await TransportStream!.WriteAsync(LastChunk, CancellationToken.None); + + //Flush base stream + await TransportStream!.FlushAsync(CancellationToken.None); + } + + protected override void Dispose(bool disposing) => Close(); + + public override void Close() + { + //If write error occured, then do not write the last chunk + if (HadError) + { + return; + } + + //Write remaining data to stream + ChunckAccumulator.Flush(TransportStream!); + + //Write final chunk + TransportStream!.Write(LastChunk.Span); + + //Flush base stream + TransportStream!.Flush(); + } + + + #region Hooks + + public void OnPrepare() + { + ChunckAccumulator.OnPrepare(); + } + + public void OnRelease() + { + ChunckAccumulator.OnRelease(); + } + + public void OnNewRequest() + { + ChunckAccumulator.OnNewRequest(); + + //Get transport stream even if not used + TransportStream = GetTransport(); + } + + public void OnComplete() + { + ChunckAccumulator.OnComplete(); + TransportStream = null; + + //Clear error flag + HadError = false; + } + + #endregion + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Response/DirectStream.cs b/Net.Http/src/Core/Response/DirectStream.cs new file mode 100644 index 0000000..957c2a6 --- /dev/null +++ b/Net.Http/src/Core/Response/DirectStream.cs @@ -0,0 +1,96 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: DirectStream.cs +* +* DirectStream.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Http.Core +{ + internal partial class HttpResponse + { + private class DirectStream : Stream + { + private Stream? BaseStream; + + public void Prepare(Stream transport) + { + BaseStream = transport; + } + + public override void Write(byte[] buffer, int offset, int count) + { + BaseStream!.Write(buffer, offset, count); + } + + public override void Write(ReadOnlySpan<byte> buffer) + { + BaseStream!.Write(buffer); + } + + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + return BaseStream!.WriteAsync(buffer, offset, count, cancellationToken); + } + + public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default) + { + return BaseStream!.WriteAsync(buffer, cancellationToken); + } + + + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new InvalidOperationException("Stream does not have a length property"); + public override long Position { get => throw new NotImplementedException(); set => throw new NotImplementedException(); } + public override int Read(byte[] buffer, int offset, int count) => throw new InvalidOperationException("Stream does not support reading"); + public override long Seek(long offset, SeekOrigin origin) => throw new InvalidOperationException("Stream does not support seeking"); + public override void SetLength(long value) => throw new InvalidOperationException("Stream does not support seeking"); + + public override void Flush() => BaseStream!.Flush(); + public override Task FlushAsync(CancellationToken cancellationToken) => BaseStream!.FlushAsync(cancellationToken); + + + public override void Close() + { + BaseStream = null; + } + + + protected override void Dispose(bool disposing) + { + //Do not call base dispose + Close(); + } + + public override ValueTask DisposeAsync() + { + Close(); + return ValueTask.CompletedTask; + } + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Response/HeaderDataAccumulator.cs b/Net.Http/src/Core/Response/HeaderDataAccumulator.cs new file mode 100644 index 0000000..715871f --- /dev/null +++ b/Net.Http/src/Core/Response/HeaderDataAccumulator.cs @@ -0,0 +1,157 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HeaderDataAccumulator.cs +* +* HeaderDataAccumulator.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Runtime.InteropServices; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; +using static VNLib.Net.Http.Core.CoreBufferHelpers; + +namespace VNLib.Net.Http.Core +{ + internal partial class HttpResponse + { + + /// <summary> + /// Specialized data accumulator for compiling response headers + /// </summary> + private class HeaderDataAccumulator : IDataAccumulator<char>, IStringSerializeable, IHttpLifeCycle + { + private readonly int BufferSize; + + public HeaderDataAccumulator(int bufferSize) + { + //Calc correct char buffer size from bin buffer + this.BufferSize = bufferSize * sizeof(char); + } + + /* + * May be an issue but wanted to avoid alloc + * if possible since this is a field in a ref + * type + */ + + private UnsafeMemoryHandle<byte>? _handle; + + public void Advance(int count) + { + //Advance writer + AccumulatedSize += count; + } + + public void WriteLine() => this.Append(HttpHelpers.CRLF); + + public void WriteLine(ReadOnlySpan<char> data) + { + this.Append(data); + WriteLine(); + } + + /*Use bin buffers and cast to char buffer*/ + private Span<char> Buffer => MemoryMarshal.Cast<byte, char>(_handle!.Value.Span); + + public int RemainingSize => Buffer.Length - AccumulatedSize; + public Span<char> Remaining => Buffer[AccumulatedSize..]; + public Span<char> Accumulated => Buffer[..AccumulatedSize]; + public int AccumulatedSize { get; set; } + + /// <summary> + /// Encodes the buffered data and writes it to the stream, + /// attemts to avoid further allocation where possible + /// </summary> + /// <param name="enc"></param> + /// <param name="baseStream"></param> + public void Flush(Encoding enc, Stream baseStream) + { + ReadOnlySpan<char> span = Accumulated; + //Calc the size of the binary buffer + int byteSize = enc.GetByteCount(span); + //See if there is enough room in the current char buffer + if (RemainingSize < (byteSize / sizeof(char))) + { + //We need to alloc a binary buffer to write data to + using UnsafeMemoryHandle<byte> bin = GetBinBuffer(byteSize, false); + //encode data + int encoded = enc.GetBytes(span, bin.Span); + //Write to stream + baseStream.Write(bin.Span[..encoded]); + } + else + { + //Get bin buffer by casting remaining accumulator buffer + Span<byte> bin = MemoryMarshal.Cast<char, byte>(Remaining); + //encode data + int encoded = enc.GetBytes(span, bin); + //Write to stream + baseStream.Write(bin[..encoded]); + } + Reset(); + } + + public void Reset() => AccumulatedSize = 0; + + + + public void OnPrepare() + { + //Alloc buffer + _handle = GetBinBuffer(BufferSize, false); + } + + public void OnRelease() + { + _handle!.Value.Dispose(); + _handle = null; + } + + public void OnNewRequest() + {} + + public void OnComplete() + { + Reset(); + } + + + ///<inheritdoc/> + public string Compile() => Accumulated.ToString(); + ///<inheritdoc/> + public void Compile(ref ForwardOnlyWriter<char> writer) => writer.Append(Accumulated); + ///<inheritdoc/> + public ERRNO Compile(in Span<char> buffer) + { + ForwardOnlyWriter<char> writer = new(buffer); + Compile(ref writer); + return writer.Written; + } + ///<inheritdoc/> + public override string ToString() => Compile(); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Response/HttpContextExtensions.cs b/Net.Http/src/Core/Response/HttpContextExtensions.cs new file mode 100644 index 0000000..12702b3 --- /dev/null +++ b/Net.Http/src/Core/Response/HttpContextExtensions.cs @@ -0,0 +1,124 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpContextExtensions.cs +* +* HttpContextExtensions.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http.Core +{ + /// <summary> + /// Provides extended funcionality of an <see cref="HttpContext"/> + /// </summary> + internal static class HttpContextExtensions + { + /// <summary> + /// Responds to a connection with the given status code + /// </summary> + /// <param name="ctx"></param> + /// <param name="code">The status code to send</param> + /// <exception cref="InvalidOperationException"></exception> + public static void Respond(this HttpContext ctx, HttpStatusCode code) => ctx.Response.SetStatusCode(code); + + /// <summary> + /// Begins a 301 redirection by sending status code and message heaaders to client. + /// </summary> + /// <param name="ctx"></param> + /// <param name="location">Location to direct client to, sets the "Location" header</param> + /// <exception cref="InvalidOperationException"></exception> + public static void Redirect301(this HttpContext ctx, Uri location) + { + ctx.Response.SetStatusCode(HttpStatusCode.MovedPermanently); + //Encode the string for propery http url formatting and set the location header + ctx.Response.Headers[HttpResponseHeader.Location] = location.ToString(); + } + + public const string NO_CACHE_STRING = "no-cache"; + private static readonly string CACHE_CONTROL_VALUE = HttpHelpers.GetCacheString(CacheType.NoCache | CacheType.NoStore); + + /// <summary> + /// Sets CacheControl and Pragma headers to no-cache + /// </summary> + /// <param name="Response"></param> + public static void SetNoCache(this HttpResponse Response) + { + Response.Headers[HttpResponseHeader.Pragma] = NO_CACHE_STRING; + Response.Headers[HttpResponseHeader.CacheControl] = CACHE_CONTROL_VALUE; + } + + /// <summary> + /// Sets the content-range header to the specified parameters + /// </summary> + /// <param name="Response"></param> + /// <param name="start">The content range start</param> + /// <param name="end">The content range end</param> + /// <param name="length">The total content length</param> + public static void SetContentRange(this HttpResponse Response, long start, long end, long length) + { + //Alloc enough space to hold the string + Span<char> buffer = stackalloc char[64]; + ForwardOnlyWriter<char> rangeBuilder = new(buffer); + //Build the range header in this format "bytes <begin>-<end>/<total>" + rangeBuilder.Append("bytes "); + rangeBuilder.Append(start); + rangeBuilder.Append('-'); + rangeBuilder.Append(end); + rangeBuilder.Append('/'); + rangeBuilder.Append(length); + //Print to a string and set the content range header + Response.Headers[HttpResponseHeader.ContentRange] = rangeBuilder.ToString(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal static ReadOnlyMemory<byte> GetRemainingConstrained(this IMemoryResponseReader reader, int limit) + { + //Calc the remaining bytes + int size = Math.Min(reader.Remaining, limit); + //get segment and slice + return reader.GetMemory()[..size]; + } + + /// <summary> + /// If an end-range is set, returns the remaining bytes up to the end-range, otherwise returns the entire request body length + /// </summary> + /// <param name="body"></param> + /// <param name="range">The data range</param> + /// <returns></returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal static long GetResponseLengthWithRange(this IHttpResponseBody body, Tuple<long, long> range) + { + /* + * If end range is defined, then calculate the length of the response + * + * The length is the end range minus the start range plus 1 because range + * is an inclusve value + */ + + return range.Item2 < 0 ? body.Length : Math.Min(body.Length, range.Item2 - range.Item1 + 1); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Response/HttpContextResponseWriting.cs b/Net.Http/src/Core/Response/HttpContextResponseWriting.cs new file mode 100644 index 0000000..b03363e --- /dev/null +++ b/Net.Http/src/Core/Response/HttpContextResponseWriting.cs @@ -0,0 +1,253 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpContextResponseWriting.cs +* +* HttpContextResponseWriting.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.IO.Compression; +using System.IO; +using System.Net; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Http.Core +{ + + internal partial class HttpContext + { + ///<inheritdoc/> + public async Task WriteResponseAsync(CancellationToken cancellation) + { + /* + * If exceptions are raised, the transport is unusable, the connection is terminated, + * and the release method will be called so the context can be reused + */ + + ValueTask discardTask = Request.InputStream.DiscardRemainingAsync(ParentServer.Config.DiscardBufferSize); + + //See if discard is needed + if (ResponseBody.HasData) + { + //Parallel the write and discard + Task response = WriteResponseInternalAsync(cancellation); + Task discard = discardTask.AsTask(); + + await Task.WhenAll(response, discard); + } + else + { + await discardTask; + } + + //Close response + await Response.CloseAsync(); + } + + /// <summary> + /// If implementing application set a response entity body, it is written to the output stream + /// </summary> + /// <param name="token">A token to cancel the operation</param> + private async Task WriteResponseInternalAsync(CancellationToken token) + { + //Adjust/append vary header + Response.Headers.Add(HttpResponseHeader.Vary, "Accept-Encoding"); + + //For head methods + if (Request.Method == HttpMethod.HEAD) + { + if (Request.Range != null) + { + //Get local range + Tuple<long, long> range = Request.Range; + + //Calc constrained content length + long length = ResponseBody.GetResponseLengthWithRange(range); + + //End range is inclusive so substract 1 + long endRange = (range.Item1 + length) - 1; + + //Set content-range header + Response.SetContentRange(range.Item1, endRange, length); + + //Specify what the content length would be + Response.Headers[HttpResponseHeader.ContentLength] = length.ToString(); + + } + else + { + //If the request method is head, do everything but send the body + Response.Headers[HttpResponseHeader.ContentLength] = ResponseBody.Length.ToString(); + } + + //We must send headers here so content length doesnt get overwritten + Response.FlushHeaders(); + } + else + { + Stream outputStream; + /* + * Process range header, data will not be compressed because that would + * require buffering, not a feature yet, and since the range will tell + * us the content length, we can get a direct stream to write to + */ + if (Request.Range != null) + { + //Get local range + Tuple<long, long> range = Request.Range; + + //Calc constrained content length + long length = ResponseBody.GetResponseLengthWithRange(range); + + //End range is inclusive so substract 1 + long endRange = (range.Item1 + length) - 1; + + //Set content-range header + Response.SetContentRange(range.Item1, endRange, length); + + //Get the raw output stream and set the length to the number of bytes + outputStream = Response.GetStream(length); + + await WriteEntityDataAsync(outputStream, length, token); + } + else + { + //Determine if compression should be used + bool compressionDisabled = + //disabled because app code disabled it + ContextFlags.IsSet(COMPRESSION_DISABLED_MSK) + //Disabled because too large or too small + || ResponseBody.Length >= ParentServer.Config.CompressionLimit + || ResponseBody.Length < ParentServer.Config.CompressionMinimum + //Disabled because lower than http11 does not support chunked encoding + || Request.HttpVersion < HttpVersion.Http11; + + //Get first compression method or none if disabled + HttpRequestExtensions.CompressionType ct = compressionDisabled ? HttpRequestExtensions.CompressionType.None : Request.GetCompressionSupport(); + + switch (ct) + { + case HttpRequestExtensions.CompressionType.Gzip: + { + //Specify gzip encoding (using chunked encoding) + Response.Headers[HttpResponseHeader.ContentEncoding] = "gzip"; + //get the chunked output stream + Stream chunked = Response.GetStream(); + //Use chunked encoding and send data as its written + outputStream = new GZipStream(chunked, ParentServer.Config.CompressionLevel, false); + } + break; + case HttpRequestExtensions.CompressionType.Deflate: + { + //Specify gzip encoding (using chunked encoding) + Response.Headers[HttpResponseHeader.ContentEncoding] = "deflate"; + //get the chunked output stream + Stream chunked = Response.GetStream(); + //Use chunked encoding and send data as its written + outputStream = new DeflateStream(chunked, ParentServer.Config.CompressionLevel, false); + } + break; + case HttpRequestExtensions.CompressionType.Brotli: + { + //Specify Brotli encoding (using chunked encoding) + Response.Headers[HttpResponseHeader.ContentEncoding] = "br"; + //get the chunked output stream + Stream chunked = Response.GetStream(); + //Use chunked encoding and send data as its written + outputStream = new BrotliStream(chunked, ParentServer.Config.CompressionLevel, false); + } + break; + //Default is no compression + case HttpRequestExtensions.CompressionType.None: + default: + //Since we know how long the response will be, we can submit it now (see note above for same issues) + outputStream = Response.GetStream(ResponseBody.Length); + break; + } + + //Write entity to output + await WriteEntityDataAsync(outputStream, token); + } + } + } + + private async Task WriteEntityDataAsync(Stream outputStream, CancellationToken token) + { + try + { + //Determine if buffer is required + if (ResponseBody.BufferRequired) + { + //Calc a buffer size (always a safe cast since rbs is an integer) + int bufferSize = (int)Math.Min((long)ParentServer.Config.ResponseBufferSize, ResponseBody.Length); + + //Alloc buffer, and dispose when completed + using IMemoryOwner<byte> buffer = CoreBufferHelpers.GetMemory(bufferSize, false); + + //Write response + await ResponseBody.WriteEntityAsync(outputStream, buffer.Memory, token); + } + //No buffer is required, write response directly + else + { + //Write without buffer + await ResponseBody.WriteEntityAsync(outputStream, null, token); + } + } + finally + { + //always dispose output stream + await outputStream.DisposeAsync(); + } + } + + private async Task WriteEntityDataAsync(Stream outputStream, long length, CancellationToken token) + { + try + { + //Determine if buffer is required + if (ResponseBody.BufferRequired) + { + //Calc a buffer size (always a safe cast since rbs is an integer) + int bufferSize = (int)Math.Min((long)ParentServer.Config.ResponseBufferSize, ResponseBody.Length); + + //Alloc buffer, and dispose when completed + using IMemoryOwner<byte> buffer = CoreBufferHelpers.GetMemory(bufferSize, false); + + //Write response + await ResponseBody.WriteEntityAsync(outputStream, length, buffer.Memory, token); + } + //No buffer is required, write response directly + else + { + //Write without buffer + await ResponseBody.WriteEntityAsync(outputStream, length, null, token); + } + } + finally + { + //always dispose output stream + await outputStream.DisposeAsync(); + } + } + } +} diff --git a/Net.Http/src/Core/Response/HttpResponse.cs b/Net.Http/src/Core/Response/HttpResponse.cs new file mode 100644 index 0000000..ab0971d --- /dev/null +++ b/Net.Http/src/Core/Response/HttpResponse.cs @@ -0,0 +1,307 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpResponse.cs +* +* HttpResponse.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Text; +using System.Threading.Tasks; +using System.Collections.Generic; +using System.Runtime.CompilerServices; + +using VNLib.Utils.IO; + +namespace VNLib.Net.Http.Core +{ + internal partial class HttpResponse : IHttpLifeCycle + { + private readonly HashSet<HttpCookie> Cookies; + private readonly HeaderDataAccumulator Writer; + + private readonly DirectStream ReusableDirectStream; + private readonly ChunkedStream ReusableChunkedStream; + private readonly Func<Stream> _getStream; + private readonly Encoding ResponseEncoding; + private readonly Func<HttpVersion> GetVersion; + + private bool HeadersSent; + private bool HeadersBegun; + + private HttpStatusCode _code; + + /// <summary> + /// Response header collection + /// </summary> + public VnWebHeaderCollection Headers { get; } + + public HttpResponse(Encoding encoding, int headerBufferSize, int chunkedBufferSize, Func<Stream> getStream, Func<HttpVersion> getVersion) + { + //Initialize a new header collection and a cookie jar + Headers = new(); + Cookies = new(); + //Create a new reusable writer stream + Writer = new(headerBufferSize); + + _getStream = getStream; + ResponseEncoding = encoding; + + //Create a new chunked stream + ReusableChunkedStream = new(encoding, chunkedBufferSize, getStream); + ReusableDirectStream = new(); + GetVersion = getVersion; + } + + + /// <summary> + /// Sets the status code of the response + /// </summary> + /// <exception cref="InvalidOperationException"></exception> + internal void SetStatusCode(HttpStatusCode code) + { + if (HeadersBegun) + { + throw new InvalidOperationException("Status code has already been sent"); + } + + _code = code; + } + + /// <summary> + /// Adds a new http-cookie to the collection + /// </summary> + /// <param name="cookie">Cookie to add</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal void AddCookie(HttpCookie cookie) => Cookies.Add(cookie); + + /// <summary> + /// Allows sending an early 100-Continue status message to the client + /// </summary> + /// <exception cref="InvalidOperationException"></exception> + internal async Task SendEarly100ContinueAsync() + { + Check(); + //Send a status message with the continue response status + Writer.WriteLine(HttpHelpers.GetResponseString(GetVersion(), HttpStatusCode.Continue)); + //Trailing crlf + Writer.WriteLine(); + //get base stream + Stream bs = _getStream(); + //Flush writer to stream (will reset the buffer) + Writer.Flush(ResponseEncoding, bs); + //Flush the base stream + await bs.FlushAsync(); + } + + /// <summary> + /// Sends the status message and all available headers to the client. + /// Headers set after method returns will be sent when output stream is requested or scope exits + /// </summary> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public void FlushHeaders() + { + Check(); + //If headers havent been sent yet, start with status code + if (!HeadersBegun) + { + //write status code first + Writer.WriteLine(HttpHelpers.GetResponseString(GetVersion(), _code)); + + //Write the date to header buffer + Writer.Append("Date: "); + Writer.Append(DateTimeOffset.UtcNow, "R"); + Writer.WriteLine(); + //Set begun flag + HeadersBegun = true; + } + //Write headers + for (int i = 0; i < Headers.Count; i++) + { + Writer.Append(Headers.Keys[i]); //Write header key + Writer.Append(": "); //Write separator + Writer.WriteLine(Headers[i]); //Write the header value + } + //Remove writen headers + Headers.Clear(); + //Write cookies if any are set + if (Cookies.Count > 0) + { + //Write cookies if any have been set + foreach (HttpCookie cookie in Cookies) + { + Writer.Append("Set-Cookie: "); + Writer.Append(in cookie); + Writer.WriteLine(); + } + //Clear all current cookies + Cookies.Clear(); + } + } + private void EndFlushHeaders(Stream transport) + { + //Sent all available headers + FlushHeaders(); + //Last line to end headers + Writer.WriteLine(); + + //Flush writer + Writer.Flush(ResponseEncoding, transport); + //Update sent headers + HeadersSent = true; + } + + /// <summary> + /// Gets a stream for writing data of a specified length directly to the client + /// </summary> + /// <param name="ContentLength"></param> + /// <returns>A <see cref="Stream"/> configured for writing data to client</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public Stream GetStream(long ContentLength) + { + Check(); + //Add content length header + Headers[HttpResponseHeader.ContentLength] = ContentLength.ToString(); + //End sending headers so the user can write to the ouput stream + Stream transport = _getStream(); + EndFlushHeaders(transport); + + //Init direct stream + ReusableDirectStream.Prepare(transport); + + //Return the direct stream + return ReusableDirectStream; + } + + /// <summary> + /// Sets up the client for chuncked encoding and gets a stream that allows for chuncks to be sent. User must call dispose on stream when done writing data + /// </summary> + /// <returns><see cref="Stream"/> supporting chunked encoding</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public Stream GetStream() + { +#if DEBUG + if (GetVersion() != HttpVersion.Http11) + { + throw new InvalidOperationException("Chunked transfer encoding is not acceptable for this http version"); + } +#endif + Check(); + //Set encoding type to chunked with user-defined compression + Headers[HttpResponseHeader.TransferEncoding] = "chunked"; + //End sending headers so the user can write to the ouput stream + Stream transport = _getStream(); + EndFlushHeaders(transport); + + //Return the reusable stream + return ReusableChunkedStream; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected void Check() + { + if (HeadersSent) + { + throw new InvalidOperationException("Headers have already been sent!"); + } + } + + /// <summary> + /// Finalzies the response to a client by sending all available headers if + /// they have not been sent yet + /// </summary> + /// <exception cref="OutOfMemoryException"></exception> + internal async ValueTask CloseAsync() + { + //If headers havent been sent yet, send them and there must be no content + if (!HeadersBegun) + { + //RFC 7230, length only set on 200 + but not 204 + if ((int)_code >= 200 && (int)_code != 204) + { + //If headers havent been sent by this stage there is no content, so set length to 0 + Headers[HttpResponseHeader.ContentLength] = "0"; + } + //Flush transport + Stream transport = _getStream(); + EndFlushHeaders(transport); + //Flush transport + await transport.FlushAsync(); + } + //Headers have been started but not finished yet + else if (!HeadersSent) + { + //RFC 7230, length only set on 200 + but not 204 + if ((int)_code >= 200 && (int)_code != 204) + { + //If headers havent been sent by this stage there is no content, so set length to 0 + Headers[HttpResponseHeader.ContentLength] = "0"; + } + //If headers arent done sending yet, conclude headers + Stream transport = _getStream(); + EndFlushHeaders(transport); + //Flush transport + await transport.FlushAsync(); + } + } + + + public void OnPrepare() + { + //Propagate all child lifecycle hooks + Writer.OnPrepare(); + ReusableChunkedStream.OnPrepare(); + } + + public void OnRelease() + { + Writer.OnRelease(); + ReusableChunkedStream.OnRelease(); + } + + public void OnNewRequest() + { + //Default to okay status code + _code = HttpStatusCode.OK; + + Writer.OnNewRequest(); + ReusableChunkedStream.OnNewRequest(); + } + + public void OnComplete() + { + //Clear headers and cookies + Headers.Clear(); + Cookies.Clear(); + //Reset status values + HeadersBegun = false; + HeadersSent = false; + + //Call child lifecycle hooks + Writer.OnComplete(); + ReusableChunkedStream.OnComplete(); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/Response/ResponseWriter.cs b/Net.Http/src/Core/Response/ResponseWriter.cs new file mode 100644 index 0000000..c9f20b5 --- /dev/null +++ b/Net.Http/src/Core/Response/ResponseWriter.cs @@ -0,0 +1,182 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ResponseWriter.cs +* +* ResponseWriter.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Extensions; + + +namespace VNLib.Net.Http.Core +{ + [System.Diagnostics.CodeAnalysis.SuppressMessage("Reliability", "CA2007:Consider calling ConfigureAwait on the awaited task", Justification = "<Pending>")] + internal sealed class ResponseWriter : IHttpResponseBody, IHttpLifeCycle + { + private Stream? _streamResponse; + private IMemoryResponseReader? _memoryResponse; + + ///<inheritdoc/> + public bool HasData { get; private set; } + + //Buffering is required when a stream is set + bool IHttpResponseBody.BufferRequired => _streamResponse != null; + + ///<inheritdoc/> + public long Length { get; private set; } + + /// <summary> + /// Attempts to set the response body as a stream + /// </summary> + /// <param name="response">The stream response body to read</param> + /// <returns>True if the response entity could be set, false if it has already been set</returns> + internal bool TrySetResponseBody(Stream response) + { + if (HasData) + { + return false; + } + + //Get relative length of the stream, IE the remaning bytes in the stream if position has been modified + Length = (response.Length - response.Position); + //Store ref to stream + _streamResponse = response; + //update has-data flag + HasData = true; + return true; + } + + /// <summary> + /// Attempts to set the response entity + /// </summary> + /// <param name="response">The memory response to set</param> + /// <returns>True if the response entity could be set, false if it has already been set</returns> + internal bool TrySetResponseBody(IMemoryResponseReader response) + { + if (HasData) + { + return false; + } + + //Get length + Length = response.Remaining; + //Store ref to stream + _memoryResponse = response; + //update has-data flag + HasData = true; + return true; + } + + ///<inheritdoc/> + async Task IHttpResponseBody.WriteEntityAsync(Stream dest, long count, Memory<byte>? buffer, CancellationToken token) + { + //Write a sliding window response + if (_memoryResponse != null) + { + //Get min value from count/range length + int remaining = (int)Math.Min(count, _memoryResponse.Remaining); + + //Write response body from memory + while (remaining > 0) + { + //Get remaining segment + ReadOnlyMemory<byte> segment = _memoryResponse.GetRemainingConstrained(remaining); + + //Write segment to output stream + await dest.WriteAsync(segment, token); + + int written = segment.Length; + + //Advance by the written ammount + _memoryResponse.Advance(written); + + //Update remaining + remaining -= written; + } + } + else + { + //Buffer is required, and count must be supplied + await _streamResponse!.CopyToAsync(dest, buffer!.Value, count, token); + } + } + + ///<inheritdoc/> + async Task IHttpResponseBody.WriteEntityAsync(Stream dest, Memory<byte>? buffer, CancellationToken token) + { + //Write a sliding window response + if (_memoryResponse != null) + { + //Write response body from memory + while (_memoryResponse.Remaining > 0) + { + //Get segment + ReadOnlyMemory<byte> segment = _memoryResponse.GetMemory(); + + await dest.WriteAsync(segment, token); + + //Advance by + _memoryResponse.Advance(segment.Length); + } + } + else + { + //Buffer is required + await _streamResponse!.CopyToAsync(dest, buffer!.Value, token); + + //Try to dispose the response stream + await _streamResponse!.DisposeAsync(); + + //remove ref + _streamResponse = null; + } + } + + ///<inheritdoc/> + void IHttpLifeCycle.OnPrepare() + {} + + ///<inheritdoc/> + void IHttpLifeCycle.OnRelease() + {} + + ///<inheritdoc/> + void IHttpLifeCycle.OnNewRequest() + {} + + public void OnComplete() + { + //Clear has data flag + HasData = false; + Length = 0; + + //Clear rseponse containers + _streamResponse?.Dispose(); + _streamResponse = null; + _memoryResponse?.Close(); + _memoryResponse = null; + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/SharedHeaderReaderBuffer.cs b/Net.Http/src/Core/SharedHeaderReaderBuffer.cs new file mode 100644 index 0000000..36ebb66 --- /dev/null +++ b/Net.Http/src/Core/SharedHeaderReaderBuffer.cs @@ -0,0 +1,85 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: SharedHeaderReaderBuffer.cs +* +* SharedHeaderReaderBuffer.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.InteropServices; + +using VNLib.Utils.Memory; + + + +namespace VNLib.Net.Http.Core +{ + sealed class SharedHeaderReaderBuffer : IHttpLifeCycle + { + private UnsafeMemoryHandle<byte>? Handle; + + /// <summary> + /// The size of the binary buffer + /// </summary> + public int BinLength { get; } + + private readonly int _bufferSize; + + internal SharedHeaderReaderBuffer(int length) + { + _bufferSize = length + (length * sizeof(char)); + + //Bin buffer is the specified size + BinLength = length; + } + + /// <summary> + /// The binary buffer to store reader information + /// </summary> + public Span<byte> BinBuffer => Handle!.Value.Span[..BinLength]; + + /// <summary> + /// The char buffer to store read characters in + /// </summary> + public Span<char> CharBuffer => MemoryMarshal.Cast<byte, char>(Handle!.Value.Span[BinLength..]); + + public void OnPrepare() + { + //Alloc the shared buffer + Handle = CoreBufferHelpers.GetBinBuffer(_bufferSize, true); + } + + public void OnRelease() + { + //Free buffer + Handle?.Dispose(); + Handle = null; + } + + public void OnNewRequest() + {} + + public void OnComplete() + { + //Zero buffer + Handle!.Value.Span.Clear(); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Core/VnHeaderCollection.cs b/Net.Http/src/Core/VnHeaderCollection.cs new file mode 100644 index 0000000..8ce3c88 --- /dev/null +++ b/Net.Http/src/Core/VnHeaderCollection.cs @@ -0,0 +1,75 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: VnHeaderCollection.cs +* +* VnHeaderCollection.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Collections.Generic; + +namespace VNLib.Net.Http.Core +{ + internal sealed class VnHeaderCollection : IHeaderCollection + { + private VnWebHeaderCollection _RequestHeaders; + private VnWebHeaderCollection _ResponseHeaders; + + + IEnumerable<KeyValuePair<string, string>> IHeaderCollection.RequestHeaders => _RequestHeaders!; + + IEnumerable<KeyValuePair<string, string>> IHeaderCollection.ResponseHeaders => _ResponseHeaders!; + + internal VnHeaderCollection(HttpContext context) + { + _RequestHeaders = context.Request.Headers; + _ResponseHeaders = context.Response.Headers; + } + + string? IHeaderCollection.this[string index] + { + get => _RequestHeaders[index]; + set => _ResponseHeaders[index] = value; + } + + string IHeaderCollection.this[HttpResponseHeader index] + { + set => _ResponseHeaders[index] = value; + } + + string? IHeaderCollection.this[HttpRequestHeader index] => _RequestHeaders[index]; + + bool IHeaderCollection.HeaderSet(HttpResponseHeader header) => !string.IsNullOrEmpty(_ResponseHeaders[header]); + + bool IHeaderCollection.HeaderSet(HttpRequestHeader header) => !string.IsNullOrEmpty(_RequestHeaders[header]); + + void IHeaderCollection.Append(HttpResponseHeader header, string? value) => _ResponseHeaders.Add(header, value); + + void IHeaderCollection.Append(string header, string? value) => _ResponseHeaders.Add(header, value); + +#nullable disable + internal void Clear() + { + _RequestHeaders = null; + _ResponseHeaders = null; + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Exceptions/ContentTypeException.cs b/Net.Http/src/Exceptions/ContentTypeException.cs new file mode 100644 index 0000000..abff151 --- /dev/null +++ b/Net.Http/src/Exceptions/ContentTypeException.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ContentTypeException.cs +* +* ContentTypeException.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Thrown when the application attempts to submit a response to a client + /// when the client does not accept the given content type + /// </summary> + public sealed class ContentTypeUnacceptableException:FormatException + { + public ContentTypeUnacceptableException(string message) : base(message) {} + + public ContentTypeUnacceptableException() + {} + + public ContentTypeUnacceptableException(string message, Exception innerException) : base(message, innerException) + {} + } +}
\ No newline at end of file diff --git a/Net.Http/src/Exceptions/TerminateConnectionException.cs b/Net.Http/src/Exceptions/TerminateConnectionException.cs new file mode 100644 index 0000000..b854b6e --- /dev/null +++ b/Net.Http/src/Exceptions/TerminateConnectionException.cs @@ -0,0 +1,57 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: TerminateConnectionException.cs +* +* TerminateConnectionException.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; + +namespace VNLib.Net.Http +{ + /// <summary> + /// User code may throw this exception to signal the <see cref="HttpServer"/> to drop + /// the transport connection and return an optional status code + /// </summary> + public class TerminateConnectionException : Exception + { + internal HttpStatusCode Code { get; } + + /// <summary> + /// Creates a new instance that terminates the connection without sending a response to the connection + /// </summary> + public TerminateConnectionException() : base(){} + /// <summary> + /// Creates a new instance of the connection exception with an error code to respond to the connection with + /// </summary> + /// <param name="responseCode">The status code to send to the user</param> + public TerminateConnectionException(HttpStatusCode responseCode) + { + this.Code = responseCode; + } + + public TerminateConnectionException(string message) : base(message) + {} + + public TerminateConnectionException(string message, Exception innerException) : base(message, innerException) + {} + } +}
\ No newline at end of file diff --git a/Net.Http/src/FileUpload.cs b/Net.Http/src/FileUpload.cs new file mode 100644 index 0000000..654d682 --- /dev/null +++ b/Net.Http/src/FileUpload.cs @@ -0,0 +1,122 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: FileUpload.cs +* +* FileUpload.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; + +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +using static VNLib.Net.Http.Core.CoreBufferHelpers; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Represents an file that was received as an entity body, either using Multipart/FormData or as the entity body itself + /// </summary> + public readonly struct FileUpload + { + /// <summary> + /// Content type of uploaded file + /// </summary> + public readonly ContentType ContentType; + /// <summary> + /// Name of file uploaded + /// </summary> + public readonly string FileName; + /// <summary> + /// The file data captured on upload + /// </summary> + public readonly Stream FileData; + + private readonly bool OwnsHandle; + + /// <summary> + /// Allocates a new binary buffer, encodes, and copies the specified data to a new <see cref="FileUpload"/> + /// structure of the specified content type + /// </summary> + /// <param name="data">The string data to copy</param> + /// <param name="dataEncoding">The encoding instance to encode the string data from</param> + /// <param name="filename">The name of the file</param> + /// <param name="ct">The content type of the file data</param> + /// <returns>The <see cref="FileUpload"/> container</returns> + internal static FileUpload FromString(ReadOnlySpan<char> data, Encoding dataEncoding, string filename, ContentType ct) + { + //get number of bytes + int bytes = dataEncoding.GetByteCount(data); + //get a buffer from the HTTP heap + MemoryHandle<byte> buffHandle = HttpPrivateHeap.Alloc<byte>(bytes); + try + { + //Convert back to binary + bytes = dataEncoding.GetBytes(data, buffHandle); + + //Create a new memory stream encapsulating the file data + VnMemoryStream vms = VnMemoryStream.ConsumeHandle(buffHandle, bytes, true); + + //Create new upload wrapper + return new (vms, filename, ct, true); + } + catch + { + //Make sure the hanle gets disposed if there is an error + buffHandle.Dispose(); + throw; + } + } + + /// <summary> + /// Initialzes a new <see cref="FileUpload"/> structure from the specified data + /// and file information. + /// </summary> + /// <param name="data"></param> + /// <param name="filename"></param> + /// <param name="ct"></param> + /// <param name="ownsHandle"></param> + public FileUpload(Stream data, string filename, ContentType ct, bool ownsHandle) + { + FileName = filename; + ContentType = ct; + //Store handle ownership + OwnsHandle = ownsHandle; + //Store the stream + FileData = data; + } + + /// <summary> + /// Releases any memory the current instance holds if it owns the handles + /// </summary> + internal readonly void Free() + { + //Dispose the handle if we own it + if (OwnsHandle) + { + //This should always be synchronous + FileData.Dispose(); + } + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Helpers/AlternateProtocolTransportStreamWrapper.cs b/Net.Http/src/Helpers/AlternateProtocolTransportStreamWrapper.cs new file mode 100644 index 0000000..88891fb --- /dev/null +++ b/Net.Http/src/Helpers/AlternateProtocolTransportStreamWrapper.cs @@ -0,0 +1,50 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: AlternateProtocolTransportStreamWrapper.cs +* +* AlternateProtocolTransportStreamWrapper.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading.Tasks; + +using VNLib.Utils.IO; + +namespace VNLib.Net.Http.Core +{ + internal class AlternateProtocolTransportStreamWrapper : BackingStream<Stream> + { + public AlternateProtocolTransportStreamWrapper(Stream transport) + { + this.BaseStream = transport; + } + + //Do not allow the caller to dispose the transport stream + protected override void Dispose(bool disposing) + {} + public override ValueTask DisposeAsync() + { + return ValueTask.CompletedTask; + } + public override void Close() + {} + } +} diff --git a/Net.Http/src/Helpers/ContentType.cs b/Net.Http/src/Helpers/ContentType.cs new file mode 100644 index 0000000..ce7b7ce --- /dev/null +++ b/Net.Http/src/Helpers/ContentType.cs @@ -0,0 +1,1180 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ContentType.cs +* +* ContentType.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +namespace VNLib.Net.Http +{ + /// <summary> + /// Mime content type + /// </summary> + public enum ContentType + { + NonSupported, + MultiPart, + UrlEncoded, + Aab, + Aac, + Aam, + Aas, + Abw, + Ac, + Acc, + Ace, + Acu, + Acutc, + Adp, + Aep, + Afm, + Afp, + Ahead, + Ai, + Aif, + Aifc, + Aiff, + Air, + Ait, + Ami, + Amr, + Apk, + Apng, + Appcache, + Apr, + Arc, + Arj, + Asc, + Asf, + Asm, + Aso, + Asx, + Atc, + Atom, + Atomcat, + Atomsvc, + Atx, + Au, + Avi, + Avif, + Aw, + Azf, + Azs, + Azv, + Azw, + B16, + Bat, + Bcpio, + Bdf, + Bdm, + Bdoc, + Bed, + Bh2, + Binary, + Blb, + Blorb, + Bmi, + Bmml, + Bmp, + Book, + Box, + Boz, + Bpk, + Bsp, + Btif, + Buffer, + Bz, + Bz2, + C, + C11amc, + C11amz, + C4d, + C4f, + C4g, + C4p, + C4u, + Cab, + Caf, + Cap, + Car, + Cat, + Cb7, + Cba, + Cbr, + Cbt, + Cbz, + Cc, + Cco, + Cct, + Ccxml, + Cdbcmsg, + Cdf, + Cdfx, + Cdkey, + Cdmia, + Cdmic, + Cdmid, + Cdmio, + Cdmiq, + Cdx, + Cdxml, + Cdy, + Cer, + Cfs, + Cgm, + Chat, + Chm, + Chrt, + Cif, + Cii, + Cil, + Cjs, + Cla, + Clkk, + Clkp, + Clkt, + Clkw, + Clkx, + Clp, + Cmc, + Cmdf, + Cml, + Cmp, + Cmx, + Cod, + Coffee, + Com, + Conf, + Cpio, + Cpp, + Cpt, + Crd, + Crl, + Crt, + Crx, + Csh, + Csl, + Csml, + Csp, + Css, + Cst, + Csv, + Cu, + Curl, + Cww, + Cxt, + Cxx, + Dae, + Daf, + Dart, + Dataless, + Davmount, + Dbf, + Dbk, + Dcr, + Dcurl, + Dd2, + Ddd, + Ddf, + Dds, + Deb, + Def, + Deploy, + Der, + Dfac, + Dgc, + Dic, + Dir, + Dis, + Dist, + Distz, + Djv, + Djvu, + Dll, + Dmg, + Dmp, + Dms, + Dna, + Doc, + Docm, + Docx, + Dot, + Dotm, + Dotx, + Dp, + Dpg, + Dra, + Drle, + Dsc, + Dssc, + Dtb, + Dtd, + Dts, + Dtshd, + Dump, + Dvb, + Dvi, + Dwd, + Dwf, + Dwg, + Dxf, + Dxp, + Dxr, + Ear, + Ecma, + Edm, + Edx, + Efif, + Ei6, + Elc, + Emf, + Eml, + Emma, + Emz, + Eol, + Eot, + Eps, + Epub, + Es, + Es3, + Esa, + Esf, + Et3, + Etx, + Eva, + Evy, + Exe, + Exi, + Exp, + Exr, + Ext, + Ez, + Ez2, + Ez3, + F, + F4v, + Fortran, + F90, + Fbs, + Fcdt, + Fcs, + Fdf, + Fdt, + Fg5, + Fgd, + Fh, + Fh4, + Fh5, + Fh7, + Fhc, + Fig, + Fits, + Flac, + Fli, + Flo, + Flv, + Flw, + Flx, + Fly, + Fm, + Fnc, + Fo, + For, + Fpx, + Frame, + Fsc, + Fst, + Ftc, + Fti, + Fvt, + Fxp, + Fxpl, + Fzs, + G2w, + G3, + G3w, + Gac, + Gam, + Gbr, + Gca, + Gdl, + Gdoc, + Geo, + Geojson, + Gex, + Ggb, + Ggt, + Ghf, + Gif, + Gim, + Glb, + Gltf, + Gml, + Gmx, + Gnumeric, + Gph, + Gpx, + Gqf, + Gqs, + Gram, + Gramps, + Gre, + Grv, + Grxml, + Gsf, + Gsheet, + Gslides, + Gtar, + Gtm, + Gtw, + Gv, + Gxf, + Gxt, + Gz, + H, + H261, + H263, + H264, + Hal, + Hbci, + Hbs, + Hdd, + Hdf, + Heic, + Heics, + Heif, + Heifs, + Hej2, + Held, + Hh, + Hjson, + Hlp, + Hpgl, + Hpid, + Hps, + Hqx, + Hsj2, + Htc, + Htke, + Htm, + Html, + Hvd, + Hvp, + Hvs, + I2g, + Icc, + Ice, + Icm, + Ico, + Ics, + Ief, + Ifb, + Ifm, + Iges, + Igl, + Igm, + Igs, + Igx, + Iif, + Img, + Imp, + Ims, + Ini, + Ink, + Inkml, + Install, + Iota, + Ipfix, + Ipk, + Irm, + Irp, + Iso, + Itp, + Its, + Ivp, + Ivu, + Jad, + Jade, + Jam, + Jar, + Jardiff, + Java, + Jhc, + Jisp, + Jls, + Jlt, + Jng, + Jnlp, + Joda, + Jp2, + Jpe, + Jpeg, + Jpf, + Jpg, + Jpg2, + Jpgm, + Jpgv, + Jph, + Jpm, + Jpx, + Javascript, + Json, + Json5, + Jsonld, + Jsonml, + Jsx, + Jxr, + Jxra, + Jxrs, + Jxs, + Jxsc, + Jxsi, + Jxss, + Kar, + Karbon, + Kdbx, + Key, + Kfo, + Kia, + Kml, + Kmz, + Kne, + Knp, + Kon, + Kpr, + Kpt, + Kpxx, + Ksp, + Ktr, + Ktx, + Ktx2, + Ktz, + Kwd, + Kwt, + Lasxml, + Latex, + Lbd, + Lbe, + Les, + Less, + Lgr, + Lha, + Link66, + List, + List3820, + Listafp, + Lnk, + Log, + Lostxml, + Lrf, + Lrm, + Ltf, + Lua, + Luac, + Lvp, + Lwp, + Lzh, + M13, + M14, + M1v, + M21, + M2a, + M2v, + M3a, + M3u, + M3u8, + M4a, + M4p, + M4s, + M4u, + M4v, + Ma, + Mads, + Maei, + Mag, + Maker, + Man, + Manifest, + Map, + Mar, + Markdown, + Mathml, + Mb, + Mbk, + Mbox, + Mc1, + Mcd, + Mcurl, + Md, + Mdb, + Mdi, + Mdx, + Me, + Mesh, + Meta4, + Metalink, + Mets, + Mfm, + Mft, + Mgp, + Mgz, + Mid, + Midi, + Mie, + Mif, + Mime, + Mj2, + Mjp2, + Mjs, + Mk3d, + Mka, + Mkd, + Mks, + Mkv, + Mlp, + Mmd, + Mmf, + Mml, + Mmr, + Mng, + Mny, + Mobi, + Mods, + Mov, + Movie, + Mp2, + Mp21, + Mp2a, + Mp3, + Mp4, + Mp4a, + Mp4s, + Mp4v, + Mpc, + Mpd, + Mpe, + Mpeg, + Mpg, + Mpg4, + Mpga, + Mpkg, + Mpm, + Mpn, + Mpp, + Mpt, + Mpy, + Mqy, + Mrc, + Mrcx, + Ms, + Mscml, + Mseed, + Mseq, + Msf, + Msg, + Msh, + Msi, + Msl, + Msm, + Msp, + Msty, + Mtl, + Mts, + Mus, + Musd, + Musicxml, + Mvb, + Mvt, + Mwf, + Mxf, + Mxl, + Mxmf, + Mxml, + Mxs, + Mxu, + N3, + Nb, + Nbp, + Nc, + Ncx, + Nfo, + Ngdat, + Nitf, + Nlu, + Nml, + Nnd, + Nns, + Nnw, + Npx, + Nq, + Nsc, + Nsf, + Nt, + Ntf, + Numbers, + Nzb, + Oa2, + Oa3, + Oas, + Obd, + Obgx, + Obj, + Oda, + Odb, + Odc, + Odf, + Odft, + Odg, + Odi, + Odm, + Odp, + Ods, + Odt, + Oga, + Ogex, + Ogg, + Ogv, + Ogx, + Omdoc, + Onepkg, + Onetmp, + Onetoc, + Onetoc2, + Opf, + Opml, + Oprc, + Opus, + Org, + Osf, + Osfpvg, + Osm, + Otc, + Otf, + Otg, + Oth, + Oti, + Otp, + Ots, + Ott, + Ova, + Ovf, + Owl, + Oxps, + Oxt, + P, + P10, + P12, + P7b, + P7c, + P7m, + P7r, + P7s, + P8, + Pac, + Pages, + Pas, + Paw, + Pbd, + Pbm, + Pcap, + Pcf, + Pcl, + Pclxl, + Pct, + Pcurl, + Pcx, + Pdb, + Pde, + Pdf, + Pem, + Pfa, + Pfb, + Pfm, + Pfr, + Pfx, + Pgm, + Pgn, + Pgp, + Php, + Pic, + Pkg, + Pki, + Pkipath, + Pkpass, + Pl, + Plb, + Plc, + Plf, + Pls, + Pm, + Pml, + Png, + Pnm, + Portpkg, + Pot, + Potm, + Potx, + Ppam, + Ppd, + Ppm, + Pps, + Ppsm, + Ppsx, + Ppt, + Pptm, + Pptx, + Pqa, + Prc, + Pre, + Prf, + Provx, + Ps, + Psb, + Psd, + Psf, + Pskcxml, + Pti, + Ptid, + Pub, + Pvb, + Pwn, + Pya, + Pyv, + Qam, + Qbo, + Qfx, + Qps, + Qt, + Qwd, + Qwt, + Qxb, + Qxd, + Qxl, + Qxt, + Ra, + Ram, + Raml, + Rapd, + Rar, + Ras, + Rdf, + Rdz, + Relo, + Rep, + Res, + Rgb, + Rif, + Rip, + Ris, + Rl, + Rlc, + Rld, + Rm, + Rmi, + Rmp, + Rms, + Rmvb, + Rnc, + Rng, + Roa, + Roff, + Rp9, + Rpm, + Rpss, + Rpst, + Rq, + Rs, + Rsat, + Rsd, + Rsheet, + Rss, + Rtf, + Rtx, + Run, + Rusd, + S, + S3m, + Saf, + Sass, + Sbml, + Sc, + Scd, + Scm, + Scq, + Scs, + Scss, + Scurl, + Sda, + Sdc, + Sdd, + Sdkd, + Sdkm, + Sdp, + Sdw, + Sea, + See, + Seed, + Sema, + Semd, + Semf, + Senmlx, + Sensmlx, + Ser, + Setpay, + Setreg, + Sfs, + Sfv, + Sgi, + Sgl, + Sgm, + Sgml, + Sh, + Shar, + Shex, + Shf, + Shtml, + Sid, + Sieve, + Sig, + Sil, + Silo, + Sis, + Sisx, + Sit, + Sitx, + Siv, + Skd, + Skm, + Skp, + Skt, + Sldm, + Sldx, + Slim, + Slm, + Sls, + Slt, + Sm, + Smf, + Smi, + Smil, + Smv, + Smzip, + Snd, + Snf, + So, + Spc, + Spdx, + Spf, + Spl, + Spot, + Spp, + Spq, + Spx, + Sql, + Src, + Srt, + Sru, + Srx, + Ssdl, + Sse, + Ssf, + Ssml, + St, + Stc, + Std, + Stf, + Sti, + Stk, + Stl, + Stpx, + Stpxz, + Stpz, + Str, + Stw, + Styl, + Stylus, + Sub, + Sus, + Susp, + Sv4cpio, + Sv4crc, + Svc, + Svd, + Svg, + Svgz, + Swa, + Swf, + Swi, + Swidtag, + Sxc, + Sxd, + Sxg, + Sxi, + Sxm, + Sxw, + T, + T3, + T38, + Taglet, + Tao, + Tap, + Tar, + Tcap, + Tcl, + Td, + Teacher, + Tei, + Tex, + Texi, + Texinfo, + Text, + Tfi, + Tfm, + Tfx, + Tga, + Thmx, + Tif, + Tiff, + Tk, + Tmo, + Toml, + Torrent, + Tpl, + Tpt, + Tr, + Tra, + Trig, + Trm, + Ts, + Tsd, + Tsv, + Ttc, + Ttf, + Ttl, + Ttml, + Twd, + Twds, + Txd, + Txf, + Txt, + U32, + U8dsn, + U8hdr, + U8mdn, + U8msg, + Ubj, + Udeb, + Ufd, + Ufdl, + Ulx, + Umj, + Unityweb, + Uoml, + Uri, + Uris, + Urls, + Usdz, + Ustar, + Utz, + Uu, + Uva, + Uvd, + Uvf, + Uvg, + Uvh, + Uvi, + Uvm, + Uvp, + Uvs, + Uvt, + Uvu, + Uvv, + Uvva, + Uvvd, + Uvvf, + Uvvg, + Uvvh, + Uvvi, + Uvvm, + Uvvp, + Uvvs, + Uvvt, + Uvvu, + Uvvv, + Uvvx, + Uvvz, + Uvx, + Uvz, + Vbox, + Vcard, + Vcd, + Vcf, + Vcg, + Vcs, + Vcx, + Vdi, + Vds, + Vhd, + Vis, + Viv, + Vmdk, + Vob, + Vor, + Vox, + Vrml, + Vsd, + Vsf, + Vss, + Vst, + Vsw, + Vtf, + Vtt, + Vtu, + Vxml, + W3d, + Wad, + Wadl, + War, + Wasm, + Wav, + Wax, + Wbmp, + Wbs, + Wbxml, + Wcm, + Wdb, + Wdp, + Weba, + Webapp, + Webm, + Webp, + Wg, + Wgt, + Wks, + Wm, + Wma, + Wmd, + Wmf, + Wml, + Wmlc, + Wmls, + Wmlsc, + Wmv, + Wmx, + Wmz, + Woff, + Woff2, + Wpd, + Wpl, + Wps, + Wqd, + Wri, + Wrl, + Wsc, + Wsdl, + Wspolicy, + Wtb, + Wvx, + X32, + X3d, + X3db, + X3dbz, + X3dv, + X3dvz, + X3dz, + Xaml, + Xap, + Xar, + Xav, + Xbap, + Xbd, + Xbm, + Xca, + Xcs, + Xdf, + Xdm, + Xdp, + Xdssc, + Xdw, + Xel, + Xenc, + Xer, + Xfdf, + Xfdl, + Xht, + Xhtml, + Xhvml, + Xif, + Xla, + Xlam, + Xlc, + Xlf, + Xlm, + Xls, + Xlsb, + Xlsm, + Xlsx, + Xlt, + Xltm, + Xltx, + Xlw, + Xm, + Xml, + Xns, + Xo, + Xop, + Xpi, + Xpl, + Xpm, + Xpr, + Xps, + Xpw, + Xpx, + Xsd, + Xsl, + Xslt, + Xsm, + Xspf, + Xul, + Xvm, + Xvml, + Xwd, + Xyz, + Xz, + Yaml, + Yang, + Yin, + Yml, + Ymp, + Z1, + Z2, + Z3, + Z4, + Z5, + Z6, + Z7, + Z8, + Zaz, + Zip, + Zir, + Zirz, + Zmm, + } +}
\ No newline at end of file diff --git a/Net.Http/src/Helpers/CoreBufferHelpers.cs b/Net.Http/src/Helpers/CoreBufferHelpers.cs new file mode 100644 index 0000000..5c5d918 --- /dev/null +++ b/Net.Http/src/Helpers/CoreBufferHelpers.cs @@ -0,0 +1,188 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: CoreBufferHelpers.cs +* +* CoreBufferHelpers.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +/* + * This class is meant to provide memory helper methods + * as a centralized HTTP local memory api. + * + * Pools and heaps are privatized to help avoid + * leaking sensitive HTTP data across other application + * allocations and help provide memory optimization. + */ + + + +using System; +using System.IO; +using System.Buffers; +using System.Security; +using System.Threading; + +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http.Core +{ + + /// <summary> + /// Provides memory pools and an internal heap for allocations. + /// </summary> + internal static class CoreBufferHelpers + { + private class InitDataBuffer : ISlindingWindowBuffer<byte> + { + private readonly ArrayPool<byte> pool; + private readonly int size; + + private byte[]? buffer; + + public InitDataBuffer(ArrayPool<byte> pool, int size) + { + this.buffer = pool.Rent(size, true); + this.pool = pool; + this.size = size; + WindowStartPos = 0; + WindowEndPos = 0; + } + + public int WindowStartPos { get; set; } + public int WindowEndPos { get; set; } + Memory<byte> ISlindingWindowBuffer<byte>.Buffer => buffer.AsMemory(0, size); + + public void Advance(int count) + { + WindowEndPos += count; + } + + public void AdvanceStart(int count) + { + WindowStartPos += count; + } + + public void Reset() + { + WindowStartPos = 0; + WindowEndPos = 0; + } + + //Release the buffer back to the pool + void ISlindingWindowBuffer<byte>.Close() + { + pool.Return(buffer!); + buffer = null; + } + } + + /// <summary> + /// An internal HTTP character binary pool for HTTP specific internal buffers + /// </summary> + public static ArrayPool<byte> HttpBinBufferPool { get; } = ArrayPool<byte>.Create(); + /// <summary> + /// An <see cref="IUnmangedHeap"/> used for internal HTTP buffers + /// </summary> + public static IUnmangedHeap HttpPrivateHeap => _lazyHeap.Value; + + private static readonly Lazy<IUnmangedHeap> _lazyHeap = new(Memory.InitializeNewHeapForProcess, LazyThreadSafetyMode.PublicationOnly); + + /// <summary> + /// Alloctes an unsafe block of memory from the internal heap, or buffer pool + /// </summary> + /// <param name="size">The number of elemnts to allocate</param> + /// <param name="zero">A value indicating of the block should be zeroed before returning</param> + /// <returns>A handle to the block of memory</returns> + /// <exception cref="SecurityException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static UnsafeMemoryHandle<byte> GetBinBuffer(int size, bool zero) + { + //Calc buffer size to the nearest page size + size = (size / 4096 + 1) * 4096; + + //If rpmalloc lib is loaded, use it + if (Memory.IsRpMallocLoaded) + { + return Memory.Shared.UnsafeAlloc<byte>(size, zero); + } + else if (size > Memory.MAX_UNSAFE_POOL_SIZE) + { + return HttpPrivateHeap.UnsafeAlloc<byte>(size, zero); + } + else + { + return new(HttpBinBufferPool, size, zero); + } + } + + public static IMemoryOwner<byte> GetMemory(int size, bool zero) + { + //Calc buffer size to the nearest page size + size = (size / 4096 + 1) * 4096; + + //If rpmalloc lib is loaded, use it + if (Memory.IsRpMallocLoaded) + { + return Memory.Shared.DirectAlloc<byte>(size, zero); + } + //Avoid locking in heap unless the buffer is too large to alloc array + else if (size > Memory.MAX_UNSAFE_POOL_SIZE) + { + return HttpPrivateHeap.DirectAlloc<byte>(size, zero); + } + else + { + //Convert temp buffer to memory owner + +#pragma warning disable CA2000 // Dispose objects before losing scope + return new VnTempBuffer<byte>(HttpBinBufferPool, size, zero).ToMemoryManager(); +#pragma warning restore CA2000 // Dispose objects before losing scope + } + } + + /// <summary> + /// Gets the remaining data in the reader buffer and prepares a + /// sliding window buffer to read data from + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="reader"></param> + /// <param name="maxContentLength">Maximum content size to clamp the remaining buffer window to</param> + /// <returns></returns> + public static ISlindingWindowBuffer<byte>? GetReminaingData<T>(this ref T reader, long maxContentLength) where T: struct, IVnTextReader + { + //clamp max available to max content length + int available = Math.Clamp(reader.Available, 0, (int)maxContentLength); + if (available <= 0) + { + return null; + } + //Alloc sliding window buffer + ISlindingWindowBuffer<byte> buffer = new InitDataBuffer(HttpBinBufferPool, available); + //Read remaining data + reader.ReadRemaining(buffer.RemainingBuffer.Span); + //Advance the buffer to the end of available data + buffer.Advance(available); + return buffer; + } + + } +} diff --git a/Net.Http/src/Helpers/HelperTypes.cs b/Net.Http/src/Helpers/HelperTypes.cs new file mode 100644 index 0000000..e89ca78 --- /dev/null +++ b/Net.Http/src/Helpers/HelperTypes.cs @@ -0,0 +1,98 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HelperTypes.cs +* +* HelperTypes.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Net.Http +{ + [Flags] + public enum HttpMethod + { + NOT_SUPPORTED, + GET = 0x01, + POST = 0x02, + PUT = 0x04, + OPTIONS = 0x08, + HEAD = 0x10, + MERGE = 0x20, + COPY = 0x40, + DELETE = 0x80, + PATCH = 0x100, + TRACE = 0x200, + MOVE = 0x400, + LOCK = 0x800 + } + /// <summary> + /// HTTP protocol version + /// </summary> + [Flags] + public enum HttpVersion + { + NotSupported, + Http1 = 0x01, + Http11 = 0x02, + Http2 = 0x04, + Http09 = 0x08 + } + /// <summary> + /// HTTP response entity cache flags + /// </summary> + [Flags] + public enum CacheType + { + Ignore = 0x00, + NoCache = 0x01, + Private = 0x02, + Public = 0x04, + NoStore = 0x08, + Revalidate = 0x10 + } + + /// <summary> + /// Specifies an HTTP cookie SameSite type + /// </summary> + public enum CookieSameSite + { + Lax, None, SameSite + } + + /// <summary> + /// Low level 301 "hard" redirect + /// </summary> + public class Redirect + { + public readonly string Url; + public readonly Uri RedirectUrl; + /// <summary> + /// Quickly redirects a url to another url before sessions are established + /// </summary> + /// <param name="url">Url to redirect on</param> + /// <param name="redirecturl">Url to redirect to</param> + public Redirect(string url, string redirecturl) + { + Url = url; + RedirectUrl = new(redirecturl); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Helpers/HttpHelpers.cs b/Net.Http/src/Helpers/HttpHelpers.cs new file mode 100644 index 0000000..9cceff1 --- /dev/null +++ b/Net.Http/src/Helpers/HttpHelpers.cs @@ -0,0 +1,445 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpHelpers.cs +* +* HttpHelpers.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Net.Sockets; +using System.Collections.Generic; +using System.Text.RegularExpressions; + +using VNLib.Net.Http.Core; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Provides a set of HTTP helper functions + /// </summary> + public static partial class HttpHelpers + { + /// <summary> + /// Carrage return + line feed characters used within the VNLib.Net.Http namespace to delimit http messages/lines + /// </summary> + public const string CRLF = "\r\n"; + + public const string WebsocketRFC4122Guid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + + public const string EVENT_STREAM_ACCEPT_TYPE = "text/event-stream"; + + /// <summary> + /// Extended <see cref="HttpRequestHeader"/> for origin header, DO NOT USE IN <see cref="WebHeaderCollection"/> + /// </summary> + internal const HttpRequestHeader Origin = (HttpRequestHeader)42; + /// <summary> + /// Extended <see cref="HttpRequestHeader"/> for Content-Disposition, DO NOT USE IN <see cref="WebHeaderCollection"/> + /// </summary> + internal const HttpRequestHeader ContentDisposition = (HttpRequestHeader)41; + + private static readonly Regex HttpRequestBuilderRegex = new("(?<=[a-z])([A-Z])", RegexOptions.Compiled); + + /* + * Provides a hashable lookup table from a method string's hashcode to output + * an HttpMethod enum value, + */ + + private static readonly IReadOnlyDictionary<int, HttpMethod> MethodHashLookup; + + /* + * Provides a constant lookup table from an MIME http request header string to a .NET + * enum value (with some extra support) + */ + + private static readonly IReadOnlyDictionary<string, HttpRequestHeader> RequestHeaderLookup = new Dictionary<string, HttpRequestHeader>() + { + {"CacheControl", HttpRequestHeader.CacheControl }, + {"Connection", HttpRequestHeader.Connection }, + {"Date", HttpRequestHeader.Date }, + {"Keep-Alive", HttpRequestHeader.KeepAlive }, + {"Pragma", HttpRequestHeader.Pragma }, + {"Trailer", HttpRequestHeader.Trailer }, + {"Transfer-Encoding", HttpRequestHeader.TransferEncoding }, + {"Upgrade", HttpRequestHeader.Upgrade }, + {"Via", HttpRequestHeader.Via }, + {"Warning", HttpRequestHeader.Warning }, + {"Allow", HttpRequestHeader.Allow }, + {"Content-Length", HttpRequestHeader.ContentLength }, + {"Content-Type", HttpRequestHeader.ContentType }, + {"Content-Encoding", HttpRequestHeader.ContentEncoding }, + {"Content-Language", HttpRequestHeader.ContentLanguage }, + {"Content-Location", HttpRequestHeader.ContentLocation }, + {"Content-Md5", HttpRequestHeader.ContentMd5 }, + {"Content-Range", HttpRequestHeader.ContentRange }, + {"Expires", HttpRequestHeader.Expires }, + {"Last-Modified", HttpRequestHeader.LastModified }, + {"Accept", HttpRequestHeader.Accept }, + {"Accept-Charset", HttpRequestHeader.AcceptCharset }, + {"Accept-Encoding", HttpRequestHeader.AcceptEncoding }, + {"Accept-Language", HttpRequestHeader.AcceptLanguage }, + {"Authorization", HttpRequestHeader.Authorization }, + {"Cookie", HttpRequestHeader.Cookie }, + {"Expect", HttpRequestHeader.Expect }, + {"From", HttpRequestHeader.From }, + {"Host", HttpRequestHeader.Host }, + {"IfMatch", HttpRequestHeader.IfMatch }, + {"If-Modified-Since", HttpRequestHeader.IfModifiedSince }, + {"If-None-Match", HttpRequestHeader.IfNoneMatch }, + {"If-Range", HttpRequestHeader.IfRange }, + {"If-Unmodified-Since", HttpRequestHeader.IfUnmodifiedSince }, + {"MaxForwards", HttpRequestHeader.MaxForwards }, + {"Proxy-Authorization", HttpRequestHeader.ProxyAuthorization }, + {"Referer", HttpRequestHeader.Referer }, + {"Range", HttpRequestHeader.Range }, + {"Te", HttpRequestHeader.Te }, + {"Translate", HttpRequestHeader.Translate }, + {"User-Agent", HttpRequestHeader.UserAgent }, + //Custom request headers + { "Content-Disposition", ContentDisposition }, + { "origin", Origin } + }; + + /* + * Provides a lookup table for request header hashcodes (that are hashed in + * the static constructor) to ouput an http request header enum value from + * a header string's hashcode (allows for spans to produce an enum value + * during request parsing) + * + */ + private static readonly IReadOnlyDictionary<int, HttpRequestHeader> RequestHeaderHashLookup; + + /* + * Provides a constant lookup table for http version hashcodes to an http + * version enum value + */ + private static readonly IReadOnlyDictionary<int, HttpVersion> VersionHashLookup = new Dictionary<int, HttpVersion>() + { + { string.GetHashCode("HTTP/0.9", StringComparison.OrdinalIgnoreCase), HttpVersion.Http09 }, + { string.GetHashCode("HTTP/1.0", StringComparison.OrdinalIgnoreCase), HttpVersion.Http1 }, + { string.GetHashCode("HTTP/1.1", StringComparison.OrdinalIgnoreCase), HttpVersion.Http11 }, + { string.GetHashCode("HTTP/2.0", StringComparison.OrdinalIgnoreCase), HttpVersion.Http2 } + }; + + + //Pre-compiled strings for all status codes for http 1, 1.1, and 2 + private static readonly IReadOnlyDictionary<HttpStatusCode, string> V1_STAUTS_CODES; + private static readonly IReadOnlyDictionary<HttpStatusCode, string> V1_1_STATUS_CODES; + private static readonly IReadOnlyDictionary<HttpStatusCode, string> V2_STAUTS_CODES; + + static HttpHelpers() + { + { + //Setup status code dict + Dictionary<HttpStatusCode, string> v1status = new(); + Dictionary<HttpStatusCode, string> v11status = new(); + Dictionary<HttpStatusCode, string> v2status = new(); + //Get all status codes + foreach (HttpStatusCode code in Enum.GetValues<HttpStatusCode>()) + { + //Use a regex to write the status code value as a string + v1status[code] = $"HTTP/1.0 {(int)code} {HttpRequestBuilderRegex.Replace(code.ToString(), " $1")}"; + v11status[code] = $"HTTP/1.1 {(int)code} {HttpRequestBuilderRegex.Replace(code.ToString(), " $1")}"; + v2status[code] = $"HTTP/2.0 {(int)code} {HttpRequestBuilderRegex.Replace(code.ToString(), " $1")}"; + } + //Store as readonly + V1_STAUTS_CODES = v1status; + V1_1_STATUS_CODES = v11status; + V2_STAUTS_CODES = v2status; + } + { + /* + * Http methods are hashed at runtime using the HttpMethod enum + * values, purley for compatability and automation + */ + Dictionary<int, HttpMethod> methods = new(); + //Add all HTTP methods + foreach (HttpMethod method in Enum.GetValues<HttpMethod>()) + { + //Exclude the not supported method + if (method == HttpMethod.NOT_SUPPORTED) + { + continue; + } + //Store method string's hashcode for faster lookups + methods[string.GetHashCode(method.ToString(), StringComparison.OrdinalIgnoreCase)] = method; + } + MethodHashLookup = methods; + } + { + /* + * Pre-compute common headers + */ + Dictionary<int, HttpRequestHeader> requestHeaderHashes = new(); + //Add all HTTP methods + foreach (string headerValue in RequestHeaderLookup.Keys) + { + //Compute the hashcode for the header value + int hashCode = string.GetHashCode(headerValue, StringComparison.OrdinalIgnoreCase); + //Store the http header enum value with the hash-code of the string of said header + requestHeaderHashes[hashCode] = RequestHeaderLookup[headerValue]; + } + RequestHeaderHashLookup = requestHeaderHashes; + } + } + + + /// <summary> + /// Returns an http formatted content type string of a specified content type + /// </summary> + /// <param name="type">Contenty type</param> + /// <returns>Http acceptable string representing a content type</returns> + /// <exception cref="KeyNotFoundException"></exception> + public static string GetContentTypeString(ContentType type) => CtToMime[type]; + /// <summary> + /// Returns the <see cref="ContentType"/> enum value from the MIME string + /// </summary> + /// <param name="type">Content type from request</param> + /// <returns><see cref="ContentType"/> of request, <see cref="ContentType.NonSupported"/> if unknown</returns> + public static ContentType GetContentType(string type) => MimeToCt.GetValueOrDefault(type, ContentType.NonSupported); + //Cache control string using mdn reference + //https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control + /// <summary> + /// Builds a Cache-Control MIME content header from the specified flags + /// </summary> + /// <param name="type">The cache type/mode</param> + /// <param name="maxAge">The max-age (time in seconds) argument</param> + /// <param name="immutable">Sets the immutable argument</param> + /// <returns>The string representation of the Cache-Control header</returns> + public static string GetCacheString(CacheType type, int maxAge = 0, bool immutable = false) + { + //Rent a buffer to write header to + Span<char> buffer = stackalloc char[128]; + //Get buffer writer for cache header + ForwardOnlyWriter<char> sb = new(buffer); + if ((type & CacheType.NoCache) > 0) + { + sb.Append("no-cache, "); + } + if ((type & CacheType.NoStore) > 0) + { + sb.Append("no-store, "); + } + if ((type & CacheType.Public) > 0) + { + sb.Append("public, "); + } + if ((type & CacheType.Private) > 0) + { + sb.Append("private, "); + } + if ((type & CacheType.Revalidate) > 0) + { + sb.Append("must-revalidate, "); + } + if (immutable) + { + sb.Append("immutable, "); + } + sb.Append("max-age="); + sb.Append(maxAge); + return sb.ToString(); + } + /// <summary> + /// Builds a Cache-Control MIME content header from the specified flags + /// </summary> + /// <param name="type">The cache type/mode</param> + /// <param name="maxAge">The max-age argument</param> + /// <param name="immutable">Sets the immutable argument</param> + /// <returns>The string representation of the Cache-Control header</returns> + public static string GetCacheString(CacheType type, TimeSpan maxAge, bool immutable = false) => GetCacheString(type, (int)maxAge.TotalSeconds, immutable); + /// <summary> + /// Returns an enum value of an httpmethod of an http request method string + /// </summary> + /// <param name="smethod">Http acceptable method type string</param> + /// <returns>Request method, <see cref="HttpMethod.NOT_SUPPORTED"/> if method is malformatted or unsupported</returns> + /// <exception cref="ArgumentNullException"></exception> + public static HttpMethod GetRequestMethod(ReadOnlySpan<char> smethod) + { + //Get the hashcode for the method "string" + int hashCode = string.GetHashCode(smethod, StringComparison.OrdinalIgnoreCase); + //run the lookup and return not supported if the method was not found + return MethodHashLookup.GetValueOrDefault(hashCode, HttpMethod.NOT_SUPPORTED); + } + /// <summary> + /// Compares the first 3 bytes of IPV4 ip address or the first 6 bytes of a IPV6. Can be used to determine if the address is local to another address + /// </summary> + /// <param name="first">Address to be compared</param> + /// <param name="other">Address to be comared to first address</param> + /// <returns>True if first 2 bytes of each address match (Big Endian)</returns> + public static bool IsLocalSubnet(this IPAddress first, IPAddress other) + { + if(first.AddressFamily != other.AddressFamily) + { + return false; + } + switch (first.AddressFamily) + { + case AddressFamily.InterNetwork: + { + //Alloc buffers 4 bytes for IPV4 + Span<byte> firstBytes = stackalloc byte[4]; + Span<byte> otherBytes = stackalloc byte[4]; + //Write address's to the buffers + if (first.TryWriteBytes(firstBytes, out _) && other.TryWriteBytes(otherBytes, out _)) + { + //Compare the first 3 bytes of the first address to the second address + return firstBytes.StartsWith(otherBytes[..3]); + } + } + break; + case AddressFamily.InterNetworkV6: + { + //Alloc buffers 8 bytes for IPV6 + Span<byte> firstBytes = stackalloc byte[8]; + Span<byte> otherBytes = stackalloc byte[8]; + //Write address's to the buffers + if (first.TryWriteBytes(firstBytes, out _) && other.TryWriteBytes(otherBytes, out _)) + { + //Compare the first 6 bytes of the first address to the second address + return firstBytes.StartsWith(otherBytes[..6]); + } + } + break; + } + return false; + } + /// <summary> + /// Selects a <see cref="ContentType"/> for a given file extension + /// </summary> + /// <param name="path">Path (including extension) of a file</param> + /// <returns><see cref="ContentType"/> of file. Returns <see cref="ContentType.Binary"/> if extension is unknown</returns> + public static ContentType GetContentTypeFromFile(ReadOnlySpan<char> path) + { + //Get the file's extension + ReadOnlySpan<char> extention = Path.GetExtension(path); + //Trim leading . + extention = extention.Trim('.'); + //If the extension is defined, perform a lookup, otherwise return the default + return ExtensionToCt.GetValueOrDefault(extention.ToString(), ContentType.Binary); + } + /// <summary> + /// Selects a runtime compiled <see cref="string"/> matching the given <see cref="HttpStatusCode"/> and <see cref="HttpVersion"/> + /// </summary> + /// <param name="version">Version of the response string</param> + /// <param name="code">Status code of the response</param> + /// <returns>The HTTP response status line matching the code and version</returns> + public static string GetResponseString(HttpVersion version, HttpStatusCode code) + { + return version switch + { + HttpVersion.Http1 => V1_STAUTS_CODES[code], + HttpVersion.Http2 => V2_STAUTS_CODES[code], + _ => V1_1_STATUS_CODES[code], + }; + } + + /// <summary> + /// Parses the mime Content-Type header value into its sub-components + /// </summary> + /// <param name="header">The Content-Type header value field</param> + /// <param name="ContentType">The mime content type field</param> + /// <param name="Charset">The mime charset</param> + /// <param name="Boundry">The multi-part form boundry parameter</param> + /// <returns>True if parsing the content type succeded, false otherwise</returns> + public static bool TryParseContentType(string header, out string? ContentType, out string? Charset, out string? Boundry) + { + try + { + //Parse content type + System.Net.Mime.ContentType ctype = new(header); + Boundry = ctype.Boundary; + Charset = ctype.CharSet; + ContentType = ctype.MediaType; + return true; + } + catch +//Disable warning for not using the exception, intended behavior +#pragma warning disable ERP022 // Unobserved exception in a generic exception handler. + { + ContentType = Charset = Boundry = null; + //Invalid content type header value + } +#pragma warning restore ERP022 // Unobserved exception in a generic exception handler. + return false; + } + + /// <summary> + /// Parses a standard HTTP Content disposition header into its sub-components, type, name, filename (optional) + /// </summary> + /// <param name="header">The buffer containing the Content-Disposition header value only</param> + /// <param name="type">The mime form type</param> + /// <param name="name">The mime name argument</param> + /// <param name="fileName">The mime filename</param> + public static void ParseDisposition(ReadOnlySpan<char> header, out string? type, out string? name, out string? fileName) + { + //First parameter should be the type argument + type = header.SliceBeforeParam(';').Trim().ToString(); + //Set defaults for name and filename + name = fileName = null; + //get the name parameter + ReadOnlySpan<char> nameSpan = header.SliceAfterParam("name=\""); + if (!nameSpan.IsEmpty) + { + //Capture the name parameter value and trim it up + name = nameSpan.SliceBeforeParam('"').Trim().ToString(); + } + //Check for the filename parameter + ReadOnlySpan<char> fileNameSpan = header.SliceAfterParam("filename=\""); + if (!fileNameSpan.IsEmpty) + { + //Capture the name parameter value and trim it up + fileName = fileNameSpan.SliceBeforeParam('"').Trim().ToString(); + } + } + + /// <summary> + /// Performs a lookup of the specified header name to get the <see cref="HttpRequestHeader"/> enum value + /// </summary> + /// <param name="requestHeaderName">The value of the HTTP request header to compute</param> + /// <returns>The <see cref="HttpRequestHeader"/> enum value of the header, or 255 if not found</returns> + internal static HttpRequestHeader GetRequestHeaderEnumFromValue(ReadOnlySpan<char> requestHeaderName) + { + //Compute the hashcode from the header name + int hashcode = string.GetHashCode(requestHeaderName, StringComparison.OrdinalIgnoreCase); + //perform lookup + return RequestHeaderHashLookup.GetValueOrDefault(hashcode, (HttpRequestHeader)255); + } + + /// <summary> + /// Gets the <see cref="HttpVersion"/> enum value from the version string + /// </summary> + /// <param name="httpVersion">The http header version string</param> + /// <returns>The <see cref="HttpVersion"/> enum value, or + /// <see cref="HttpVersion.NotSupported"/> if the version could not be + /// determined + /// </returns> + public static HttpVersion ParseHttpVersion(ReadOnlySpan<char> httpVersion) + { + //Get the hashcode for the http version "string" + int hashCode = string.GetHashCode(httpVersion.Trim(), StringComparison.OrdinalIgnoreCase); + //return the version that matches the hashcode, or return unsupported of not found + return VersionHashLookup.GetValueOrDefault(hashCode, HttpVersion.NotSupported); + } + } +}
\ No newline at end of file diff --git a/Net.Http/src/Helpers/MimeLookups.cs b/Net.Http/src/Helpers/MimeLookups.cs new file mode 100644 index 0000000..03bc59d --- /dev/null +++ b/Net.Http/src/Helpers/MimeLookups.cs @@ -0,0 +1,3237 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: MimeLookups.cs +* +* MimeLookups.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; + +namespace VNLib.Net.Http +{ + public static partial class HttpHelpers + { + //Content type lookup dict + private static readonly IReadOnlyDictionary<ContentType, string> CtToMime = new Dictionary<ContentType, string>() + { + { ContentType.NonSupported, "application/octet-stream" }, + { ContentType.UrlEncoded, "application/x-www-form-urlencoded" }, + { ContentType.MultiPart, "multipart/form-data" }, + { ContentType.Aab, "application/x-authorware-bin" }, + { ContentType.Aac, "audio/x-aac" }, + { ContentType.Aam, "application/x-authorware-map" }, + { ContentType.Aas, "application/x-authorware-seg" }, + { ContentType.Abw, "application/x-abiword" }, + { ContentType.Ac, "application/pkix-attr-cert" }, + { ContentType.Acc, "application/vnd.americandynamics.acc" }, + { ContentType.Ace, "application/x-ace-compressed" }, + { ContentType.Acu, "application/vnd.acucobol" }, + { ContentType.Acutc, "application/vnd.acucorp" }, + { ContentType.Adp, "audio/adpcm" }, + { ContentType.Aep, "application/vnd.audiograph" }, + { ContentType.Afm, "application/x-font-type1" }, + { ContentType.Afp, "application/vnd.ibm.modcap" }, + { ContentType.Ahead, "application/vnd.ahead.space" }, + { ContentType.Ai, "application/postscript" }, + { ContentType.Aif, "audio/x-aiff" }, + { ContentType.Aifc, "audio/x-aiff" }, + { ContentType.Aiff, "audio/x-aiff" }, + { ContentType.Air, "application/vnd.adobe.air-application-installer-package+zip" }, + { ContentType.Ait, "application/vnd.dvb.ait" }, + { ContentType.Ami, "application/vnd.amiga.ami" }, + { ContentType.Amr, "audio/amr" }, + { ContentType.Apk, "application/vnd.android.package-archive" }, + { ContentType.Apng, "image/apng" }, + { ContentType.Appcache, "text/cache-manifest" }, + { ContentType.Apr, "application/vnd.lotus-approach" }, + { ContentType.Arc, "application/x-freearc" }, + { ContentType.Arj, "application/x-arj" }, + { ContentType.Asc, "application/pgp-signature" }, + { ContentType.Asf, "video/x-ms-asf" }, + { ContentType.Asm, "text/x-asm" }, + { ContentType.Aso, "application/vnd.accpac.simply.aso" }, + { ContentType.Asx, "video/x-ms-asf" }, + { ContentType.Atc, "application/vnd.acucorp" }, + { ContentType.Atom, "application/atom+xml" }, + { ContentType.Atomcat, "application/atomcat+xml" }, + { ContentType.Atomsvc, "application/atomsvc+xml" }, + { ContentType.Atx, "application/vnd.antix.game-component" }, + { ContentType.Au, "audio/basic" }, + { ContentType.Avi, "video/x-msvideo" }, + { ContentType.Avif, "image/avif" }, + { ContentType.Aw, "application/applixware" }, + { ContentType.Azf, "application/vnd.airzip.filesecure.azf" }, + { ContentType.Azs, "application/vnd.airzip.filesecure.azs" }, + { ContentType.Azv, "image/vnd.airzip.accelerator.azv" }, + { ContentType.Azw, "application/vnd.amazon.ebook" }, + { ContentType.B16, "image/vnd.pco.b16" }, + { ContentType.Bat, "application/x-msdownload" }, + { ContentType.Bcpio, "application/x-bcpio" }, + { ContentType.Bdf, "application/x-font-bdf" }, + { ContentType.Bdm, "application/vnd.syncml.dm+wbxml" }, + { ContentType.Bdoc, "application/bdoc" }, + { ContentType.Bed, "application/vnd.realvnc.bed" }, + { ContentType.Bh2, "application/vnd.fujitsu.oasysprs" }, + { ContentType.Binary, "application/octet-stream" }, + { ContentType.Blb, "application/x-blorb" }, + { ContentType.Blorb, "application/x-blorb" }, + { ContentType.Bmi, "application/vnd.bmi" }, + { ContentType.Bmml, "application/vnd.balsamiq.bmml+xml" }, + { ContentType.Bmp, "image/bmp" }, + { ContentType.Book, "application/vnd.framemaker" }, + { ContentType.Box, "application/vnd.previewsystems.box" }, + { ContentType.Boz, "application/x-bzip2" }, + { ContentType.Bpk, "application/octet-stream" }, + { ContentType.Bsp, "model/vnd.valve.source.compiled-map" }, + { ContentType.Btif, "image/prs.btif" }, + { ContentType.Buffer, "application/octet-stream" }, + { ContentType.Bz, "application/x-bzip" }, + { ContentType.Bz2, "application/x-bzip2" }, + { ContentType.C, "text/x-c" }, + { ContentType.C11amc, "application/vnd.cluetrust.cartomobile-config" }, + { ContentType.C11amz, "application/vnd.cluetrust.cartomobile-config-pkg" }, + { ContentType.C4d, "application/vnd.clonk.c4group" }, + { ContentType.C4f, "application/vnd.clonk.c4group" }, + { ContentType.C4g, "application/vnd.clonk.c4group" }, + { ContentType.C4p, "application/vnd.clonk.c4group" }, + { ContentType.C4u, "application/vnd.clonk.c4group" }, + { ContentType.Cab, "application/vnd.ms-cab-compressed" }, + { ContentType.Caf, "audio/x-caf" }, + { ContentType.Cap, "application/vnd.tcpdump.pcap" }, + { ContentType.Car, "application/vnd.curl.car" }, + { ContentType.Cat, "application/vnd.ms-pki.seccat" }, + { ContentType.Cb7, "application/x-cbr" }, + { ContentType.Cba, "application/x-cbr" }, + { ContentType.Cbr, "application/x-cbr" }, + { ContentType.Cbt, "application/x-cbr" }, + { ContentType.Cbz, "application/x-cbr" }, + { ContentType.Cc, "text/x-c" }, + { ContentType.Cco, "application/x-cocoa" }, + { ContentType.Cct, "application/x-director" }, + { ContentType.Ccxml, "application/ccxml+xml" }, + { ContentType.Cdbcmsg, "application/vnd.contact.cmsg" }, + { ContentType.Cdf, "application/x-netcdf" }, + { ContentType.Cdfx, "application/cdfx+xml" }, + { ContentType.Cdkey, "application/vnd.mediastation.cdkey" }, + { ContentType.Cdmia, "application/cdmi-capability" }, + { ContentType.Cdmic, "application/cdmi-container" }, + { ContentType.Cdmid, "application/cdmi-domain" }, + { ContentType.Cdmio, "application/cdmi-object" }, + { ContentType.Cdmiq, "application/cdmi-queue" }, + { ContentType.Cdx, "chemical/x-cdx" }, + { ContentType.Cdxml, "application/vnd.chemdraw+xml" }, + { ContentType.Cdy, "application/vnd.cinderella" }, + { ContentType.Cer, "application/pkix-cert" }, + { ContentType.Cfs, "application/x-cfs-compressed" }, + { ContentType.Cgm, "image/cgm" }, + { ContentType.Chat, "application/x-chat" }, + { ContentType.Chm, "application/vnd.ms-htmlhelp" }, + { ContentType.Chrt, "application/vnd.kde.kchart" }, + { ContentType.Cif, "chemical/x-cif" }, + { ContentType.Cii, "application/vnd.anser-web-certificate-issue-initiation" }, + { ContentType.Cil, "application/vnd.ms-artgalry" }, + { ContentType.Cjs, "application/node" }, + { ContentType.Cla, "application/vnd.claymore" }, + { ContentType.Clkk, "application/vnd.crick.clicker.keyboard" }, + { ContentType.Clkp, "application/vnd.crick.clicker.palette" }, + { ContentType.Clkt, "application/vnd.crick.clicker.template" }, + { ContentType.Clkw, "application/vnd.crick.clicker.wordbank" }, + { ContentType.Clkx, "application/vnd.crick.clicker" }, + { ContentType.Clp, "application/x-msclip" }, + { ContentType.Cmc, "application/vnd.cosmocaller" }, + { ContentType.Cmdf, "chemical/x-cmdf" }, + { ContentType.Cml, "chemical/x-cml" }, + { ContentType.Cmp, "application/vnd.yellowriver-custom-menu" }, + { ContentType.Cmx, "image/x-cmx" }, + { ContentType.Cod, "application/vnd.rim.cod" }, + { ContentType.Coffee, "text/coffeescript" }, + { ContentType.Com, "application/x-msdownload" }, + { ContentType.Conf, "text/plain" }, + { ContentType.Cpio, "application/x-cpio" }, + { ContentType.Cpp, "text/x-c" }, + { ContentType.Cpt, "application/mac-compactpro" }, + { ContentType.Crd, "application/x-mscardfile" }, + { ContentType.Crl, "application/pkix-crl" }, + { ContentType.Crt, "application/x-x509-ca-cert" }, + { ContentType.Crx, "application/x-chrome-extension" }, + { ContentType.Csh, "application/x-csh" }, + { ContentType.Csl, "application/vnd.citationstyles.style+xml" }, + { ContentType.Csml, "chemical/x-csml" }, + { ContentType.Csp, "application/vnd.commonspace" }, + { ContentType.Css, "text/css" }, + { ContentType.Cst, "application/x-director" }, + { ContentType.Csv, "text/csv" }, + { ContentType.Cu, "application/cu-seeme" }, + { ContentType.Curl, "text/vnd.curl" }, + { ContentType.Cww, "application/prs.cww" }, + { ContentType.Cxt, "application/x-director" }, + { ContentType.Cxx, "text/x-c" }, + { ContentType.Dae, "model/vnd.collada+xml" }, + { ContentType.Daf, "application/vnd.mobius.daf" }, + { ContentType.Dart, "application/vnd.dart" }, + { ContentType.Dataless, "application/vnd.fdsn.seed" }, + { ContentType.Davmount, "application/davmount+xml" }, + { ContentType.Dbf, "application/vnd.dbf" }, + { ContentType.Dbk, "application/docbook+xml" }, + { ContentType.Dcr, "application/x-director" }, + { ContentType.Dcurl, "text/vnd.curl.dcurl" }, + { ContentType.Dd2, "application/vnd.oma.dd2+xml" }, + { ContentType.Ddd, "application/vnd.fujixerox.ddd" }, + { ContentType.Ddf, "application/vnd.syncml.dmddf+xml" }, + { ContentType.Dds, "image/vnd.ms-dds" }, + { ContentType.Deb, "application/octet-stream" }, + { ContentType.Def, "text/plain" }, + { ContentType.Deploy, "application/octet-stream" }, + { ContentType.Der, "application/x-x509-ca-cert" }, + { ContentType.Dfac, "application/vnd.dreamfactory" }, + { ContentType.Dgc, "application/x-dgc-compressed" }, + { ContentType.Dic, "text/x-c" }, + { ContentType.Dir, "application/x-director" }, + { ContentType.Dis, "application/vnd.mobius.dis" }, + { ContentType.Dist, "application/octet-stream" }, + { ContentType.Distz, "application/octet-stream" }, + { ContentType.Djv, "image/vnd.djvu" }, + { ContentType.Djvu, "image/vnd.djvu" }, + { ContentType.Dll, "application/octet-stream" }, + { ContentType.Dmg, "application/octet-stream" }, + { ContentType.Dmp, "application/vnd.tcpdump.pcap" }, + { ContentType.Dms, "application/octet-stream" }, + { ContentType.Dna, "application/vnd.dna" }, + { ContentType.Doc, "application/msword" }, + { ContentType.Docm, "application/vnd.ms-word.document.macroenabled.12" }, + { ContentType.Docx, "application/vnd.openxmlformats-officedocument.wordprocessingml.document" }, + { ContentType.Dot, "application/msword" }, + { ContentType.Dotm, "application/vnd.ms-word.template.macroenabled.12" }, + { ContentType.Dotx, "application/vnd.openxmlformats-officedocument.wordprocessingml.template" }, + { ContentType.Dp, "application/vnd.osgi.dp" }, + { ContentType.Dpg, "application/vnd.dpgraph" }, + { ContentType.Dra, "audio/vnd.dra" }, + { ContentType.Drle, "image/dicom-rle" }, + { ContentType.Dsc, "text/prs.lines.tag" }, + { ContentType.Dssc, "application/dssc+der" }, + { ContentType.Dtb, "application/x-dtbook+xml" }, + { ContentType.Dtd, "application/xml-dtd" }, + { ContentType.Dts, "audio/vnd.dts" }, + { ContentType.Dtshd, "audio/vnd.dts.hd" }, + { ContentType.Dump, "application/octet-stream" }, + { ContentType.Dvb, "video/vnd.dvb.file" }, + { ContentType.Dvi, "application/x-dvi" }, + { ContentType.Dwd, "application/atsc-dwd+xml" }, + { ContentType.Dwf, "model/vnd.dwf" }, + { ContentType.Dwg, "image/vnd.dwg" }, + { ContentType.Dxf, "image/vnd.dxf" }, + { ContentType.Dxp, "application/vnd.spotfire.dxp" }, + { ContentType.Dxr, "application/x-director" }, + { ContentType.Ear, "application/java-archive" }, + { ContentType.Ecma, "application/ecmascript" }, + { ContentType.Edm, "application/vnd.novadigm.edm" }, + { ContentType.Edx, "application/vnd.novadigm.edx" }, + { ContentType.Efif, "application/vnd.picsel" }, + { ContentType.Ei6, "application/vnd.pg.osasli" }, + { ContentType.Elc, "application/octet-stream" }, + { ContentType.Emf, "application/x-msmetafile" }, + { ContentType.Eml, "message/rfc822" }, + { ContentType.Emma, "application/emma+xml" }, + { ContentType.Emz, "application/x-msmetafile" }, + { ContentType.Eol, "audio/vnd.digital-winds" }, + { ContentType.Eot, "application/vnd.ms-fontobject" }, + { ContentType.Eps, "application/postscript" }, + { ContentType.Epub, "application/epub+zip" }, + { ContentType.Es, "application/ecmascript" }, + { ContentType.Es3, "application/vnd.eszigno3+xml" }, + { ContentType.Esa, "application/vnd.osgi.subsystem" }, + { ContentType.Esf, "application/vnd.epson.esf" }, + { ContentType.Et3, "application/vnd.eszigno3+xml" }, + { ContentType.Etx, "text/x-setext" }, + { ContentType.Eva, "application/x-eva" }, + { ContentType.Evy, "application/x-envoy" }, + { ContentType.Exe, "application/octet-stream" }, + { ContentType.Exi, "application/exi" }, + { ContentType.Exp, "application/express" }, + { ContentType.Exr, "image/aces" }, + { ContentType.Ext, "application/vnd.novadigm.ext" }, + { ContentType.Ez, "application/andrew-inset" }, + { ContentType.Ez2, "application/vnd.ezpix-album" }, + { ContentType.Ez3, "application/vnd.ezpix-package" }, + { ContentType.F, "text/x-fortran" }, + { ContentType.F4v, "video/x-f4v" }, + { ContentType.Fortran, "text/x-fortran" }, + { ContentType.F90, "text/x-fortran" }, + { ContentType.Fbs, "image/vnd.fastbidsheet" }, + { ContentType.Fcdt, "application/vnd.adobe.formscentral.fcdt" }, + { ContentType.Fcs, "application/vnd.isac.fcs" }, + { ContentType.Fdf, "application/vnd.fdf" }, + { ContentType.Fdt, "application/fdt+xml" }, + { ContentType.Fg5, "application/vnd.fujitsu.oasysgp" }, + { ContentType.Fgd, "application/x-director" }, + { ContentType.Fh, "image/x-freehand" }, + { ContentType.Fh4, "image/x-freehand" }, + { ContentType.Fh5, "image/x-freehand" }, + { ContentType.Fh7, "image/x-freehand" }, + { ContentType.Fhc, "image/x-freehand" }, + { ContentType.Fig, "application/x-xfig" }, + { ContentType.Fits, "image/fits" }, + { ContentType.Flac, "audio/x-flac" }, + { ContentType.Fli, "video/x-fli" }, + { ContentType.Flo, "application/vnd.micrografx.flo" }, + { ContentType.Flv, "video/x-flv" }, + { ContentType.Flw, "application/vnd.kde.kivio" }, + { ContentType.Flx, "text/vnd.fmi.flexstor" }, + { ContentType.Fly, "text/vnd.fly" }, + { ContentType.Fm, "application/vnd.framemaker" }, + { ContentType.Fnc, "application/vnd.frogans.fnc" }, + { ContentType.Fo, "application/vnd.software602.filler.form+xml" }, + { ContentType.For, "text/x-fortran" }, + { ContentType.Fpx, "image/vnd.fpx" }, + { ContentType.Frame, "application/vnd.framemaker" }, + { ContentType.Fsc, "application/vnd.fsc.weblaunch" }, + { ContentType.Fst, "image/vnd.fst" }, + { ContentType.Ftc, "application/vnd.fluxtime.clip" }, + { ContentType.Fti, "application/vnd.anser-web-funds-transfer-initiation" }, + { ContentType.Fvt, "video/vnd.fvt" }, + { ContentType.Fxp, "application/vnd.adobe.fxp" }, + { ContentType.Fxpl, "application/vnd.adobe.fxp" }, + { ContentType.Fzs, "application/vnd.fuzzysheet" }, + { ContentType.G2w, "application/vnd.geoplan" }, + { ContentType.G3, "image/g3fax" }, + { ContentType.G3w, "application/vnd.geospace" }, + { ContentType.Gac, "application/vnd.groove-account" }, + { ContentType.Gam, "application/x-tads" }, + { ContentType.Gbr, "application/rpki-ghostbusters" }, + { ContentType.Gca, "application/x-gca-compressed" }, + { ContentType.Gdl, "model/vnd.gdl" }, + { ContentType.Gdoc, "application/vnd.google-apps.document" }, + { ContentType.Geo, "application/vnd.dynageo" }, + { ContentType.Geojson, "application/geo+json" }, + { ContentType.Gex, "application/vnd.geometry-explorer" }, + { ContentType.Ggb, "application/vnd.geogebra.file" }, + { ContentType.Ggt, "application/vnd.geogebra.tool" }, + { ContentType.Ghf, "application/vnd.groove-help" }, + { ContentType.Gif, "image/gif" }, + { ContentType.Gim, "application/vnd.groove-identity-message" }, + { ContentType.Glb, "model/gltf-binary" }, + { ContentType.Gltf, "model/gltf+json" }, + { ContentType.Gml, "application/gml+xml" }, + { ContentType.Gmx, "application/vnd.gmx" }, + { ContentType.Gnumeric, "application/x-gnumeric" }, + { ContentType.Gph, "application/vnd.flographit" }, + { ContentType.Gpx, "application/gpx+xml" }, + { ContentType.Gqf, "application/vnd.grafeq" }, + { ContentType.Gqs, "application/vnd.grafeq" }, + { ContentType.Gram, "application/srgs" }, + { ContentType.Gramps, "application/x-gramps-xml" }, + { ContentType.Gre, "application/vnd.geometry-explorer" }, + { ContentType.Grv, "application/vnd.groove-injector" }, + { ContentType.Grxml, "application/srgs+xml" }, + { ContentType.Gsf, "application/x-font-ghostscript" }, + { ContentType.Gsheet, "application/vnd.google-apps.spreadsheet" }, + { ContentType.Gslides, "application/vnd.google-apps.presentation" }, + { ContentType.Gtar, "application/x-gtar" }, + { ContentType.Gtm, "application/vnd.groove-tool-message" }, + { ContentType.Gtw, "model/vnd.gtw" }, + { ContentType.Gv, "text/vnd.graphviz" }, + { ContentType.Gxf, "application/gxf" }, + { ContentType.Gxt, "application/vnd.geonext" }, + { ContentType.Gz, "application/gzip" }, + { ContentType.H, "text/x-c" }, + { ContentType.H261, "video/h261" }, + { ContentType.H263, "video/h263" }, + { ContentType.H264, "video/h264" }, + { ContentType.Hal, "application/vnd.hal+xml" }, + { ContentType.Hbci, "application/vnd.hbci" }, + { ContentType.Hbs, "text/x-handlebars-template" }, + { ContentType.Hdd, "application/x-virtualbox-hdd" }, + { ContentType.Hdf, "application/x-hdf" }, + { ContentType.Heic, "image/heic" }, + { ContentType.Heics, "image/heic-sequence" }, + { ContentType.Heif, "image/heif" }, + { ContentType.Heifs, "image/heif-sequence" }, + { ContentType.Hej2, "image/hej2k" }, + { ContentType.Held, "application/atsc-held+xml" }, + { ContentType.Hh, "text/x-c" }, + { ContentType.Hjson, "application/hjson" }, + { ContentType.Hlp, "application/winhlp" }, + { ContentType.Hpgl, "application/vnd.hp-hpgl" }, + { ContentType.Hpid, "application/vnd.hp-hpid" }, + { ContentType.Hps, "application/vnd.hp-hps" }, + { ContentType.Hqx, "application/mac-binhex40" }, + { ContentType.Hsj2, "image/hsj2" }, + { ContentType.Htc, "text/x-component" }, + { ContentType.Htke, "application/vnd.kenameaapp" }, + { ContentType.Htm, "text/html" }, + { ContentType.Html, "text/html" }, + { ContentType.Hvd, "application/vnd.yamaha.hv-dic" }, + { ContentType.Hvp, "application/vnd.yamaha.hv-voice" }, + { ContentType.Hvs, "application/vnd.yamaha.hv-script" }, + { ContentType.I2g, "application/vnd.intergeo" }, + { ContentType.Icc, "application/vnd.iccprofile" }, + { ContentType.Ice, "x-conference/x-cooltalk" }, + { ContentType.Icm, "application/vnd.iccprofile" }, + { ContentType.Ico, "image/vnd.microsoft.icon" }, + { ContentType.Ics, "text/calendar" }, + { ContentType.Ief, "image/ief" }, + { ContentType.Ifb, "text/calendar" }, + { ContentType.Ifm, "application/vnd.shana.informed.formdata" }, + { ContentType.Iges, "model/iges" }, + { ContentType.Igl, "application/vnd.igloader" }, + { ContentType.Igm, "application/vnd.insors.igm" }, + { ContentType.Igs, "model/iges" }, + { ContentType.Igx, "application/vnd.micrografx.igx" }, + { ContentType.Iif, "application/vnd.shana.informed.interchange" }, + { ContentType.Img, "application/octet-stream" }, + { ContentType.Imp, "application/vnd.accpac.simply.imp" }, + { ContentType.Ims, "application/vnd.ms-ims" }, + { ContentType.Ini, "text/plain" }, + { ContentType.Ink, "application/inkml+xml" }, + { ContentType.Inkml, "application/inkml+xml" }, + { ContentType.Install, "application/x-install-instructions" }, + { ContentType.Iota, "application/vnd.astraea-software.iota" }, + { ContentType.Ipfix, "application/ipfix" }, + { ContentType.Ipk, "application/vnd.shana.informed.package" }, + { ContentType.Irm, "application/vnd.ibm.rights-management" }, + { ContentType.Irp, "application/vnd.irepository.package+xml" }, + { ContentType.Iso, "application/octet-stream" }, + { ContentType.Itp, "application/vnd.shana.informed.formtemplate" }, + { ContentType.Its, "application/its+xml" }, + { ContentType.Ivp, "application/vnd.immervision-ivp" }, + { ContentType.Ivu, "application/vnd.immervision-ivu" }, + { ContentType.Jad, "text/vnd.sun.j2me.app-descriptor" }, + { ContentType.Jade, "text/jade" }, + { ContentType.Jam, "application/vnd.jam" }, + { ContentType.Jar, "application/java-archive" }, + { ContentType.Jardiff, "application/x-java-archive-diff" }, + { ContentType.Java, "text/x-java-source" }, + { ContentType.Jhc, "image/jphc" }, + { ContentType.Jisp, "application/vnd.jisp" }, + { ContentType.Jls, "image/jls" }, + { ContentType.Jlt, "application/vnd.hp-jlyt" }, + { ContentType.Jng, "image/x-jng" }, + { ContentType.Jnlp, "application/x-java-jnlp-file" }, + { ContentType.Joda, "application/vnd.joost.joda-archive" }, + { ContentType.Jp2, "image/jp2" }, + { ContentType.Jpe, "image/jpeg" }, + { ContentType.Jpeg, "image/jpeg" }, + { ContentType.Jpf, "image/jpx" }, + { ContentType.Jpg, "image/jpeg" }, + { ContentType.Jpg2, "image/jp2" }, + { ContentType.Jpgm, "video/jpm" }, + { ContentType.Jpgv, "video/jpeg" }, + { ContentType.Jph, "image/jph" }, + { ContentType.Jpm, "image/jpm" }, + { ContentType.Jpx, "image/jpx" }, + { ContentType.Javascript, "application/javascript" }, + { ContentType.Json, "application/json" }, + { ContentType.Json5, "application/json5" }, + { ContentType.Jsonld, "application/ld+json" }, + { ContentType.Jsonml, "application/jsonml+json" }, + { ContentType.Jsx, "text/jsx" }, + { ContentType.Jxr, "image/jxr" }, + { ContentType.Jxra, "image/jxra" }, + { ContentType.Jxrs, "image/jxrs" }, + { ContentType.Jxs, "image/jxs" }, + { ContentType.Jxsc, "image/jxsc" }, + { ContentType.Jxsi, "image/jxsi" }, + { ContentType.Jxss, "image/jxss" }, + { ContentType.Kar, "audio/midi" }, + { ContentType.Karbon, "application/vnd.kde.karbon" }, + { ContentType.Kdbx, "application/x-keepass2" }, + { ContentType.Key, "application/vnd.apple.keynote" }, + { ContentType.Kfo, "application/vnd.kde.kformula" }, + { ContentType.Kia, "application/vnd.kidspiration" }, + { ContentType.Kml, "application/vnd.google-earth.kml+xml" }, + { ContentType.Kmz, "application/vnd.google-earth.kmz" }, + { ContentType.Kne, "application/vnd.kinar" }, + { ContentType.Knp, "application/vnd.kinar" }, + { ContentType.Kon, "application/vnd.kde.kontour" }, + { ContentType.Kpr, "application/vnd.kde.kpresenter" }, + { ContentType.Kpt, "application/vnd.kde.kpresenter" }, + { ContentType.Kpxx, "application/vnd.ds-keypoint" }, + { ContentType.Ksp, "application/vnd.kde.kspread" }, + { ContentType.Ktr, "application/vnd.kahootz" }, + { ContentType.Ktx, "image/ktx" }, + { ContentType.Ktx2, "image/ktx2" }, + { ContentType.Ktz, "application/vnd.kahootz" }, + { ContentType.Kwd, "application/vnd.kde.kword" }, + { ContentType.Kwt, "application/vnd.kde.kword" }, + { ContentType.Lasxml, "application/vnd.las.las+xml" }, + { ContentType.Latex, "application/x-latex" }, + { ContentType.Lbd, "application/vnd.llamagraphics.life-balance.desktop" }, + { ContentType.Lbe, "application/vnd.llamagraphics.life-balance.exchange+xml" }, + { ContentType.Les, "application/vnd.hhe.lesson-player" }, + { ContentType.Less, "text/less" }, + { ContentType.Lgr, "application/lgr+xml" }, + { ContentType.Lha, "application/x-lzh-compressed" }, + { ContentType.Link66, "application/vnd.route66.link66+xml" }, + { ContentType.List, "text/plain" }, + { ContentType.List3820, "application/vnd.ibm.modcap" }, + { ContentType.Listafp, "application/vnd.ibm.modcap" }, + { ContentType.Lnk, "application/x-ms-shortcut" }, + { ContentType.Log, "text/plain" }, + { ContentType.Lostxml, "application/lost+xml" }, + { ContentType.Lrf, "application/octet-stream" }, + { ContentType.Lrm, "application/vnd.ms-lrm" }, + { ContentType.Ltf, "application/vnd.frogans.ltf" }, + { ContentType.Lua, "text/x-lua" }, + { ContentType.Luac, "application/x-lua-bytecode" }, + { ContentType.Lvp, "audio/vnd.lucent.voice" }, + { ContentType.Lwp, "application/vnd.lotus-wordpro" }, + { ContentType.Lzh, "application/x-lzh-compressed" }, + { ContentType.M13, "application/x-msmediaview" }, + { ContentType.M14, "application/x-msmediaview" }, + { ContentType.M1v, "video/mpeg" }, + { ContentType.M21, "application/mp21" }, + { ContentType.M2a, "audio/mpeg" }, + { ContentType.M2v, "video/mpeg" }, + { ContentType.M3a, "audio/mpeg" }, + { ContentType.M3u, "audio/x-mpegurl" }, + { ContentType.M3u8, "application/vnd.apple.mpegurl" }, + { ContentType.M4a, "audio/mp4" }, + { ContentType.M4p, "application/mp4" }, + { ContentType.M4s, "video/iso.segment" }, + { ContentType.M4u, "video/vnd.mpegurl" }, + { ContentType.M4v, "video/x-m4v" }, + { ContentType.Ma, "application/mathematica" }, + { ContentType.Mads, "application/mads+xml" }, + { ContentType.Maei, "application/mmt-aei+xml" }, + { ContentType.Mag, "application/vnd.ecowin.chart" }, + { ContentType.Maker, "application/vnd.framemaker" }, + { ContentType.Man, "text/troff" }, + { ContentType.Manifest, "text/cache-manifest" }, + { ContentType.Map, "application/json" }, + { ContentType.Mar, "application/octet-stream" }, + { ContentType.Markdown, "text/markdown" }, + { ContentType.Mathml, "application/mathml+xml" }, + { ContentType.Mb, "application/mathematica" }, + { ContentType.Mbk, "application/vnd.mobius.mbk" }, + { ContentType.Mbox, "application/mbox" }, + { ContentType.Mc1, "application/vnd.medcalcdata" }, + { ContentType.Mcd, "application/vnd.mcd" }, + { ContentType.Mcurl, "text/vnd.curl.mcurl" }, + { ContentType.Md, "text/markdown" }, + { ContentType.Mdb, "application/x-msaccess" }, + { ContentType.Mdi, "image/vnd.ms-modi" }, + { ContentType.Mdx, "text/mdx" }, + { ContentType.Me, "text/troff" }, + { ContentType.Mesh, "model/mesh" }, + { ContentType.Meta4, "application/metalink4+xml" }, + { ContentType.Metalink, "application/metalink+xml" }, + { ContentType.Mets, "application/mets+xml" }, + { ContentType.Mfm, "application/vnd.mfmp" }, + { ContentType.Mft, "application/rpki-manifest" }, + { ContentType.Mgp, "application/vnd.osgeo.mapguide.package" }, + { ContentType.Mgz, "application/vnd.proteus.magazine" }, + { ContentType.Mid, "audio/midi" }, + { ContentType.Midi, "audio/midi" }, + { ContentType.Mie, "application/x-mie" }, + { ContentType.Mif, "application/vnd.mif" }, + { ContentType.Mime, "message/rfc822" }, + { ContentType.Mj2, "video/mj2" }, + { ContentType.Mjp2, "video/mj2" }, + { ContentType.Mjs, "application/javascript" }, + { ContentType.Mk3d, "video/x-matroska" }, + { ContentType.Mka, "audio/x-matroska" }, + { ContentType.Mkd, "text/x-markdown" }, + { ContentType.Mks, "video/x-matroska" }, + { ContentType.Mkv, "video/x-matroska" }, + { ContentType.Mlp, "application/vnd.dolby.mlp" }, + { ContentType.Mmd, "application/vnd.chipnuts.karaoke-mmd" }, + { ContentType.Mmf, "application/vnd.smaf" }, + { ContentType.Mml, "text/mathml" }, + { ContentType.Mmr, "image/vnd.fujixerox.edmics-mmr" }, + { ContentType.Mng, "video/x-mng" }, + { ContentType.Mny, "application/x-msmoney" }, + { ContentType.Mobi, "application/x-mobipocket-ebook" }, + { ContentType.Mods, "application/mods+xml" }, + { ContentType.Mov, "video/quicktime" }, + { ContentType.Movie, "video/x-sgi-movie" }, + { ContentType.Mp2, "audio/mpeg" }, + { ContentType.Mp21, "application/mp21" }, + { ContentType.Mp2a, "audio/mpeg" }, + { ContentType.Mp3, "audio/mp3" }, + { ContentType.Mp4, "video/mp4" }, + { ContentType.Mp4a, "audio/mp4" }, + { ContentType.Mp4s, "application/mp4" }, + { ContentType.Mp4v, "video/mp4" }, + { ContentType.Mpc, "application/vnd.mophun.certificate" }, + { ContentType.Mpd, "application/dash+xml" }, + { ContentType.Mpe, "video/mpeg" }, + { ContentType.Mpeg, "video/mpeg" }, + { ContentType.Mpg, "video/mpeg" }, + { ContentType.Mpg4, "video/mp4" }, + { ContentType.Mpga, "audio/mpeg" }, + { ContentType.Mpkg, "application/vnd.apple.installer+xml" }, + { ContentType.Mpm, "application/vnd.blueice.multipass" }, + { ContentType.Mpn, "application/vnd.mophun.application" }, + { ContentType.Mpp, "application/vnd.ms-project" }, + { ContentType.Mpt, "application/vnd.ms-project" }, + { ContentType.Mpy, "application/vnd.ibm.minipay" }, + { ContentType.Mqy, "application/vnd.mobius.mqy" }, + { ContentType.Mrc, "application/marc" }, + { ContentType.Mrcx, "application/marcxml+xml" }, + { ContentType.Ms, "text/troff" }, + { ContentType.Mscml, "application/mediaservercontrol+xml" }, + { ContentType.Mseed, "application/vnd.fdsn.mseed" }, + { ContentType.Mseq, "application/vnd.mseq" }, + { ContentType.Msf, "application/vnd.epson.msf" }, + { ContentType.Msg, "application/vnd.ms-outlook" }, + { ContentType.Msh, "model/mesh" }, + { ContentType.Msi, "application/octet-stream" }, + { ContentType.Msl, "application/vnd.mobius.msl" }, + { ContentType.Msm, "application/octet-stream" }, + { ContentType.Msp, "application/octet-stream" }, + { ContentType.Msty, "application/vnd.muvee.style" }, + { ContentType.Mtl, "model/mtl" }, + { ContentType.Mts, "model/vnd.mts" }, + { ContentType.Mus, "application/vnd.musician" }, + { ContentType.Musd, "application/mmt-usd+xml" }, + { ContentType.Musicxml, "application/vnd.recordare.musicxml+xml" }, + { ContentType.Mvb, "application/x-msmediaview" }, + { ContentType.Mvt, "application/vnd.mapbox-vector-tile" }, + { ContentType.Mwf, "application/vnd.mfer" }, + { ContentType.Mxf, "application/mxf" }, + { ContentType.Mxl, "application/vnd.recordare.musicxml" }, + { ContentType.Mxmf, "audio/mobile-xmf" }, + { ContentType.Mxml, "application/xv+xml" }, + { ContentType.Mxs, "application/vnd.triscape.mxs" }, + { ContentType.Mxu, "video/vnd.mpegurl" }, + { ContentType.N3, "text/n3" }, + { ContentType.Nb, "application/mathematica" }, + { ContentType.Nbp, "application/vnd.wolfram.player" }, + { ContentType.Nc, "application/x-netcdf" }, + { ContentType.Ncx, "application/x-dtbncx+xml" }, + { ContentType.Nfo, "text/x-nfo" }, + { ContentType.Ngdat, "application/vnd.nokia.n-gage.data" }, + { ContentType.Nitf, "application/vnd.nitf" }, + { ContentType.Nlu, "application/vnd.neurolanguage.nlu" }, + { ContentType.Nml, "application/vnd.enliven" }, + { ContentType.Nnd, "application/vnd.noblenet-directory" }, + { ContentType.Nns, "application/vnd.noblenet-sealer" }, + { ContentType.Nnw, "application/vnd.noblenet-web" }, + { ContentType.Npx, "image/vnd.net-fpx" }, + { ContentType.Nq, "application/n-quads" }, + { ContentType.Nsc, "application/x-conference" }, + { ContentType.Nsf, "application/vnd.lotus-notes" }, + { ContentType.Nt, "application/n-triples" }, + { ContentType.Ntf, "application/vnd.nitf" }, + { ContentType.Numbers, "application/vnd.apple.numbers" }, + { ContentType.Nzb, "application/x-nzb" }, + { ContentType.Oa2, "application/vnd.fujitsu.oasys2" }, + { ContentType.Oa3, "application/vnd.fujitsu.oasys3" }, + { ContentType.Oas, "application/vnd.fujitsu.oasys" }, + { ContentType.Obd, "application/x-msbinder" }, + { ContentType.Obgx, "application/vnd.openblox.game+xml" }, + { ContentType.Obj, "application/x-tgif" }, + { ContentType.Oda, "application/oda" }, + { ContentType.Odb, "application/vnd.oasis.opendocument.database" }, + { ContentType.Odc, "application/vnd.oasis.opendocument.chart" }, + { ContentType.Odf, "application/vnd.oasis.opendocument.formula" }, + { ContentType.Odft, "application/vnd.oasis.opendocument.formula-template" }, + { ContentType.Odg, "application/vnd.oasis.opendocument.graphics" }, + { ContentType.Odi, "application/vnd.oasis.opendocument.image" }, + { ContentType.Odm, "application/vnd.oasis.opendocument.text-master" }, + { ContentType.Odp, "application/vnd.oasis.opendocument.presentation" }, + { ContentType.Ods, "application/vnd.oasis.opendocument.spreadsheet" }, + { ContentType.Odt, "application/vnd.oasis.opendocument.text" }, + { ContentType.Oga, "audio/ogg" }, + { ContentType.Ogex, "model/vnd.opengex" }, + { ContentType.Ogg, "audio/ogg" }, + { ContentType.Ogv, "video/ogg" }, + { ContentType.Ogx, "application/ogg" }, + { ContentType.Omdoc, "application/omdoc+xml" }, + { ContentType.Onepkg, "application/onenote" }, + { ContentType.Onetmp, "application/onenote" }, + { ContentType.Onetoc, "application/onenote" }, + { ContentType.Onetoc2, "application/onenote" }, + { ContentType.Opf, "application/oebps-package+xml" }, + { ContentType.Opml, "text/x-opml" }, + { ContentType.Oprc, "application/vnd.palm" }, + { ContentType.Opus, "audio/ogg" }, + { ContentType.Org, "application/vnd.lotus-organizer" }, + { ContentType.Osf, "application/vnd.yamaha.openscoreformat" }, + { ContentType.Osfpvg, "application/vnd.yamaha.openscoreformat.osfpvg+xml" }, + { ContentType.Osm, "application/vnd.openstreetmap.data+xml" }, + { ContentType.Otc, "application/vnd.oasis.opendocument.chart-template" }, + { ContentType.Otf, "font/otf" }, + { ContentType.Otg, "application/vnd.oasis.opendocument.graphics-template" }, + { ContentType.Oth, "application/vnd.oasis.opendocument.text-web" }, + { ContentType.Oti, "application/vnd.oasis.opendocument.image-template" }, + { ContentType.Otp, "application/vnd.oasis.opendocument.presentation-template" }, + { ContentType.Ots, "application/vnd.oasis.opendocument.spreadsheet-template" }, + { ContentType.Ott, "application/vnd.oasis.opendocument.text-template" }, + { ContentType.Ova, "application/x-virtualbox-ova" }, + { ContentType.Ovf, "application/x-virtualbox-ovf" }, + { ContentType.Owl, "application/rdf+xml" }, + { ContentType.Oxps, "application/oxps" }, + { ContentType.Oxt, "application/vnd.openofficeorg.extension" }, + { ContentType.P, "text/x-pascal" }, + { ContentType.P10, "application/pkcs10" }, + { ContentType.P12, "application/x-pkcs12" }, + { ContentType.P7b, "application/x-pkcs7-certificates" }, + { ContentType.P7c, "application/pkcs7-mime" }, + { ContentType.P7m, "application/pkcs7-mime" }, + { ContentType.P7r, "application/x-pkcs7-certreqresp" }, + { ContentType.P7s, "application/pkcs7-signature" }, + { ContentType.P8, "application/pkcs8" }, + { ContentType.Pac, "application/x-ns-proxy-autoconfig" }, + { ContentType.Pages, "application/vnd.apple.pages" }, + { ContentType.Pas, "text/x-pascal" }, + { ContentType.Paw, "application/vnd.pawaafile" }, + { ContentType.Pbd, "application/vnd.powerbuilder6" }, + { ContentType.Pbm, "image/x-portable-bitmap" }, + { ContentType.Pcap, "application/vnd.tcpdump.pcap" }, + { ContentType.Pcf, "application/x-font-pcf" }, + { ContentType.Pcl, "application/vnd.hp-pcl" }, + { ContentType.Pclxl, "application/vnd.hp-pclxl" }, + { ContentType.Pct, "image/x-pict" }, + { ContentType.Pcurl, "application/vnd.curl.pcurl" }, + { ContentType.Pcx, "image/vnd.zbrush.pcx" }, + { ContentType.Pdb, "application/vnd.palm" }, + { ContentType.Pde, "text/x-processing" }, + { ContentType.Pdf, "application/pdf" }, + { ContentType.Pem, "application/x-x509-ca-cert" }, + { ContentType.Pfa, "application/x-font-type1" }, + { ContentType.Pfb, "application/x-font-type1" }, + { ContentType.Pfm, "application/x-font-type1" }, + { ContentType.Pfr, "application/font-tdpfr" }, + { ContentType.Pfx, "application/x-pkcs12" }, + { ContentType.Pgm, "image/x-portable-graymap" }, + { ContentType.Pgn, "application/x-chess-pgn" }, + { ContentType.Pgp, "application/pgp-encrypted" }, + { ContentType.Php, "application/x-httpd-php" }, + { ContentType.Pic, "image/x-pict" }, + { ContentType.Pkg, "application/octet-stream" }, + { ContentType.Pki, "application/pkixcmp" }, + { ContentType.Pkipath, "application/pkix-pkipath" }, + { ContentType.Pkpass, "application/vnd.apple.pkpass" }, + { ContentType.Pl, "application/x-perl" }, + { ContentType.Plb, "application/vnd.3gpp.pic-bw-large" }, + { ContentType.Plc, "application/vnd.mobius.plc" }, + { ContentType.Plf, "application/vnd.pocketlearn" }, + { ContentType.Pls, "application/pls+xml" }, + { ContentType.Pm, "application/x-perl" }, + { ContentType.Pml, "application/vnd.ctc-posml" }, + { ContentType.Png, "image/png" }, + { ContentType.Pnm, "image/x-portable-anymap" }, + { ContentType.Portpkg, "application/vnd.macports.portpkg" }, + { ContentType.Pot, "application/vnd.ms-powerpoint" }, + { ContentType.Potm, "application/vnd.ms-powerpoint.template.macroenabled.12" }, + { ContentType.Potx, "application/vnd.openxmlformats-officedocument.presentationml.template" }, + { ContentType.Ppam, "application/vnd.ms-powerpoint.addin.macroenabled.12" }, + { ContentType.Ppd, "application/vnd.cups-ppd" }, + { ContentType.Ppm, "image/x-portable-pixmap" }, + { ContentType.Pps, "application/vnd.ms-powerpoint" }, + { ContentType.Ppsm, "application/vnd.ms-powerpoint.slideshow.macroenabled.12" }, + { ContentType.Ppsx, "application/vnd.openxmlformats-officedocument.presentationml.slideshow" }, + { ContentType.Ppt, "application/vnd.ms-powerpoint" }, + { ContentType.Pptm, "application/vnd.ms-powerpoint.presentation.macroenabled.12" }, + { ContentType.Pptx, "application/vnd.openxmlformats-officedocument.presentationml.presentation" }, + { ContentType.Pqa, "application/vnd.palm" }, + { ContentType.Prc, "application/x-mobipocket-ebook" }, + { ContentType.Pre, "application/vnd.lotus-freelance" }, + { ContentType.Prf, "application/pics-rules" }, + { ContentType.Provx, "application/provenance+xml" }, + { ContentType.Ps, "application/postscript" }, + { ContentType.Psb, "application/vnd.3gpp.pic-bw-small" }, + { ContentType.Psd, "image/vnd.adobe.photoshop" }, + { ContentType.Psf, "application/x-font-linux-psf" }, + { ContentType.Pskcxml, "application/pskc+xml" }, + { ContentType.Pti, "image/prs.pti" }, + { ContentType.Ptid, "application/vnd.pvi.ptid1" }, + { ContentType.Pub, "application/x-mspublisher" }, + { ContentType.Pvb, "application/vnd.3gpp.pic-bw-var" }, + { ContentType.Pwn, "application/vnd.3m.post-it-notes" }, + { ContentType.Pya, "audio/vnd.ms-playready.media.pya" }, + { ContentType.Pyv, "video/vnd.ms-playready.media.pyv" }, + { ContentType.Qam, "application/vnd.epson.quickanime" }, + { ContentType.Qbo, "application/vnd.intu.qbo" }, + { ContentType.Qfx, "application/vnd.intu.qfx" }, + { ContentType.Qps, "application/vnd.publishare-delta-tree" }, + { ContentType.Qt, "video/quicktime" }, + { ContentType.Qwd, "application/vnd.quark.quarkxpress" }, + { ContentType.Qwt, "application/vnd.quark.quarkxpress" }, + { ContentType.Qxb, "application/vnd.quark.quarkxpress" }, + { ContentType.Qxd, "application/vnd.quark.quarkxpress" }, + { ContentType.Qxl, "application/vnd.quark.quarkxpress" }, + { ContentType.Qxt, "application/vnd.quark.quarkxpress" }, + { ContentType.Ra, "audio/x-pn-realaudio" }, + { ContentType.Ram, "audio/x-pn-realaudio" }, + { ContentType.Raml, "application/raml+yaml" }, + { ContentType.Rapd, "application/route-apd+xml" }, + { ContentType.Rar, "application/vnd.rar" }, + { ContentType.Ras, "image/x-cmu-raster" }, + { ContentType.Rdf, "application/rdf+xml" }, + { ContentType.Rdz, "application/vnd.data-vision.rdz" }, + { ContentType.Relo, "application/p2p-overlay+xml" }, + { ContentType.Rep, "application/vnd.businessobjects" }, + { ContentType.Res, "application/x-dtbresource+xml" }, + { ContentType.Rgb, "image/x-rgb" }, + { ContentType.Rif, "application/reginfo+xml" }, + { ContentType.Rip, "audio/vnd.rip" }, + { ContentType.Ris, "application/x-research-info-systems" }, + { ContentType.Rl, "application/resource-lists+xml" }, + { ContentType.Rlc, "image/vnd.fujixerox.edmics-rlc" }, + { ContentType.Rld, "application/resource-lists-diff+xml" }, + { ContentType.Rm, "application/vnd.rn-realmedia" }, + { ContentType.Rmi, "audio/midi" }, + { ContentType.Rmp, "audio/x-pn-realaudio-plugin" }, + { ContentType.Rms, "application/vnd.jcp.javame.midlet-rms" }, + { ContentType.Rmvb, "application/vnd.rn-realmedia-vbr" }, + { ContentType.Rnc, "application/relax-ng-compact-syntax" }, + { ContentType.Rng, "application/xml" }, + { ContentType.Roa, "application/rpki-roa" }, + { ContentType.Roff, "text/troff" }, + { ContentType.Rp9, "application/vnd.cloanto.rp9" }, + { ContentType.Rpm, "application/x-redhat-package-manager" }, + { ContentType.Rpss, "application/vnd.nokia.radio-presets" }, + { ContentType.Rpst, "application/vnd.nokia.radio-preset" }, + { ContentType.Rq, "application/sparql-query" }, + { ContentType.Rs, "application/rls-services+xml" }, + { ContentType.Rsat, "application/atsc-rsat+xml" }, + { ContentType.Rsd, "application/rsd+xml" }, + { ContentType.Rsheet, "application/urc-ressheet+xml" }, + { ContentType.Rss, "application/rss+xml" }, + { ContentType.Rtf, "application/rtf" }, + { ContentType.Rtx, "text/richtext" }, + { ContentType.Run, "application/x-makeself" }, + { ContentType.Rusd, "application/route-usd+xml" }, + { ContentType.S, "text/x-asm" }, + { ContentType.S3m, "audio/s3m" }, + { ContentType.Saf, "application/vnd.yamaha.smaf-audio" }, + { ContentType.Sass, "text/x-sass" }, + { ContentType.Sbml, "application/sbml+xml" }, + { ContentType.Sc, "application/vnd.ibm.secure-container" }, + { ContentType.Scd, "application/x-msschedule" }, + { ContentType.Scm, "application/vnd.lotus-screencam" }, + { ContentType.Scq, "application/scvp-cv-request" }, + { ContentType.Scs, "application/scvp-cv-response" }, + { ContentType.Scss, "text/x-scss" }, + { ContentType.Scurl, "text/vnd.curl.scurl" }, + { ContentType.Sda, "application/vnd.stardivision.draw" }, + { ContentType.Sdc, "application/vnd.stardivision.calc" }, + { ContentType.Sdd, "application/vnd.stardivision.impress" }, + { ContentType.Sdkd, "application/vnd.solent.sdkm+xml" }, + { ContentType.Sdkm, "application/vnd.solent.sdkm+xml" }, + { ContentType.Sdp, "application/sdp" }, + { ContentType.Sdw, "application/vnd.stardivision.writer" }, + { ContentType.Sea, "application/x-sea" }, + { ContentType.See, "application/vnd.seemail" }, + { ContentType.Seed, "application/vnd.fdsn.seed" }, + { ContentType.Sema, "application/vnd.sema" }, + { ContentType.Semd, "application/vnd.semd" }, + { ContentType.Semf, "application/vnd.semf" }, + { ContentType.Senmlx, "application/senml+xml" }, + { ContentType.Sensmlx, "application/sensml+xml" }, + { ContentType.Ser, "application/java-serialized-object" }, + { ContentType.Setpay, "application/set-payment-initiation" }, + { ContentType.Setreg, "application/set-registration-initiation" }, + { ContentType.Sfs, "application/vnd.spotfire.sfs" }, + { ContentType.Sfv, "text/x-sfv" }, + { ContentType.Sgi, "image/sgi" }, + { ContentType.Sgl, "application/vnd.stardivision.writer-global" }, + { ContentType.Sgm, "text/sgml" }, + { ContentType.Sgml, "text/sgml" }, + { ContentType.Sh, "application/x-sh" }, + { ContentType.Shar, "application/x-shar" }, + { ContentType.Shex, "text/shex" }, + { ContentType.Shf, "application/shf+xml" }, + { ContentType.Shtml, "text/html" }, + { ContentType.Sid, "image/x-mrsid-image" }, + { ContentType.Sieve, "application/sieve" }, + { ContentType.Sig, "application/pgp-signature" }, + { ContentType.Sil, "audio/silk" }, + { ContentType.Silo, "model/mesh" }, + { ContentType.Sis, "application/vnd.symbian.install" }, + { ContentType.Sisx, "application/vnd.symbian.install" }, + { ContentType.Sit, "application/x-stuffit" }, + { ContentType.Sitx, "application/x-stuffitx" }, + { ContentType.Siv, "application/sieve" }, + { ContentType.Skd, "application/vnd.koan" }, + { ContentType.Skm, "application/vnd.koan" }, + { ContentType.Skp, "application/vnd.koan" }, + { ContentType.Skt, "application/vnd.koan" }, + { ContentType.Sldm, "application/vnd.ms-powerpoint.slide.macroenabled.12" }, + { ContentType.Sldx, "application/vnd.openxmlformats-officedocument.presentationml.slide" }, + { ContentType.Slim, "text/slim" }, + { ContentType.Slm, "text/slim" }, + { ContentType.Sls, "application/route-s-tsid+xml" }, + { ContentType.Slt, "application/vnd.epson.salt" }, + { ContentType.Sm, "application/vnd.stepmania.stepchart" }, + { ContentType.Smf, "application/vnd.stardivision.math" }, + { ContentType.Smi, "application/smil+xml" }, + { ContentType.Smil, "application/smil+xml" }, + { ContentType.Smv, "video/x-smv" }, + { ContentType.Smzip, "application/vnd.stepmania.package" }, + { ContentType.Snd, "audio/basic" }, + { ContentType.Snf, "application/x-font-snf" }, + { ContentType.So, "application/octet-stream" }, + { ContentType.Spc, "application/x-pkcs7-certificates" }, + { ContentType.Spdx, "text/spdx" }, + { ContentType.Spf, "application/vnd.yamaha.smaf-phrase" }, + { ContentType.Spl, "application/x-futuresplash" }, + { ContentType.Spot, "text/vnd.in3d.spot" }, + { ContentType.Spp, "application/scvp-vp-response" }, + { ContentType.Spq, "application/scvp-vp-request" }, + { ContentType.Spx, "audio/ogg" }, + { ContentType.Sql, "application/x-sql" }, + { ContentType.Src, "application/x-wais-source" }, + { ContentType.Srt, "application/x-subrip" }, + { ContentType.Sru, "application/sru+xml" }, + { ContentType.Srx, "application/sparql-results+xml" }, + { ContentType.Ssdl, "application/ssdl+xml" }, + { ContentType.Sse, "application/vnd.kodak-descriptor" }, + { ContentType.Ssf, "application/vnd.epson.ssf" }, + { ContentType.Ssml, "application/ssml+xml" }, + { ContentType.St, "application/vnd.sailingtracker.track" }, + { ContentType.Stc, "application/vnd.sun.xml.calc.template" }, + { ContentType.Std, "application/vnd.sun.xml.draw.template" }, + { ContentType.Stf, "application/vnd.wt.stf" }, + { ContentType.Sti, "application/vnd.sun.xml.impress.template" }, + { ContentType.Stk, "application/hyperstudio" }, + { ContentType.Stl, "application/vnd.ms-pki.stl" }, + { ContentType.Stpx, "model/step+xml" }, + { ContentType.Stpxz, "model/step-xml+zip" }, + { ContentType.Stpz, "model/step+zip" }, + { ContentType.Str, "application/vnd.pg.format" }, + { ContentType.Stw, "application/vnd.sun.xml.writer.template" }, + { ContentType.Styl, "text/stylus" }, + { ContentType.Stylus, "text/stylus" }, + { ContentType.Sub, "image/vnd.dvb.subtitle" }, + { ContentType.Sus, "application/vnd.sus-calendar" }, + { ContentType.Susp, "application/vnd.sus-calendar" }, + { ContentType.Sv4cpio, "application/x-sv4cpio" }, + { ContentType.Sv4crc, "application/x-sv4crc" }, + { ContentType.Svc, "application/vnd.dvb.service" }, + { ContentType.Svd, "application/vnd.svd" }, + { ContentType.Svg, "image/svg+xml" }, + { ContentType.Svgz, "image/svg+xml" }, + { ContentType.Swa, "application/x-director" }, + { ContentType.Swf, "application/x-shockwave-flash" }, + { ContentType.Swi, "application/vnd.aristanetworks.swi" }, + { ContentType.Swidtag, "application/swid+xml" }, + { ContentType.Sxc, "application/vnd.sun.xml.calc" }, + { ContentType.Sxd, "application/vnd.sun.xml.draw" }, + { ContentType.Sxg, "application/vnd.sun.xml.writer.global" }, + { ContentType.Sxi, "application/vnd.sun.xml.impress" }, + { ContentType.Sxm, "application/vnd.sun.xml.math" }, + { ContentType.Sxw, "application/vnd.sun.xml.writer" }, + { ContentType.T, "text/troff" }, + { ContentType.T3, "application/x-t3vm-image" }, + { ContentType.T38, "image/t38" }, + { ContentType.Taglet, "application/vnd.mynfc" }, + { ContentType.Tao, "application/vnd.tao.intent-module-archive" }, + { ContentType.Tap, "image/vnd.tencent.tap" }, + { ContentType.Tar, "application/x-tar" }, + { ContentType.Tcap, "application/vnd.3gpp2.tcap" }, + { ContentType.Tcl, "application/x-tcl" }, + { ContentType.Td, "application/urc-targetdesc+xml" }, + { ContentType.Teacher, "application/vnd.smart.teacher" }, + { ContentType.Tei, "application/tei+xml" }, + { ContentType.Tex, "application/x-tex" }, + { ContentType.Texi, "application/x-texinfo" }, + { ContentType.Texinfo, "application/x-texinfo" }, + { ContentType.Text, "text/plain" }, + { ContentType.Tfi, "application/thraud+xml" }, + { ContentType.Tfm, "application/x-tex-tfm" }, + { ContentType.Tfx, "image/tiff-fx" }, + { ContentType.Tga, "image/x-tga" }, + { ContentType.Thmx, "application/vnd.ms-officetheme" }, + { ContentType.Tif, "image/tiff" }, + { ContentType.Tiff, "image/tiff" }, + { ContentType.Tk, "application/x-tcl" }, + { ContentType.Tmo, "application/vnd.tmobile-livetv" }, + { ContentType.Toml, "application/toml" }, + { ContentType.Torrent, "application/x-bittorrent" }, + { ContentType.Tpl, "application/vnd.groove-tool-template" }, + { ContentType.Tpt, "application/vnd.trid.tpt" }, + { ContentType.Tr, "text/troff" }, + { ContentType.Tra, "application/vnd.trueapp" }, + { ContentType.Trig, "application/trig" }, + { ContentType.Trm, "application/x-msterminal" }, + { ContentType.Ts, "video/mp2t" }, + { ContentType.Tsd, "application/timestamped-data" }, + { ContentType.Tsv, "text/tab-separated-values" }, + { ContentType.Ttc, "font/collection" }, + { ContentType.Ttf, "font/ttf" }, + { ContentType.Ttl, "text/turtle" }, + { ContentType.Ttml, "application/ttml+xml" }, + { ContentType.Twd, "application/vnd.simtech-mindmapper" }, + { ContentType.Twds, "application/vnd.simtech-mindmapper" }, + { ContentType.Txd, "application/vnd.genomatix.tuxedo" }, + { ContentType.Txf, "application/vnd.mobius.txf" }, + { ContentType.Txt, "text/plain" }, + { ContentType.U32, "application/x-authorware-bin" }, + { ContentType.U8dsn, "message/global-delivery-status" }, + { ContentType.U8hdr, "message/global-headers" }, + { ContentType.U8mdn, "message/global-disposition-notification" }, + { ContentType.U8msg, "message/global" }, + { ContentType.Ubj, "application/ubjson" }, + { ContentType.Udeb, "application/x-debian-package" }, + { ContentType.Ufd, "application/vnd.ufdl" }, + { ContentType.Ufdl, "application/vnd.ufdl" }, + { ContentType.Ulx, "application/x-glulx" }, + { ContentType.Umj, "application/vnd.umajin" }, + { ContentType.Unityweb, "application/vnd.unity" }, + { ContentType.Uoml, "application/vnd.uoml+xml" }, + { ContentType.Uri, "text/uri-list" }, + { ContentType.Uris, "text/uri-list" }, + { ContentType.Urls, "text/uri-list" }, + { ContentType.Usdz, "model/vnd.usdz+zip" }, + { ContentType.Ustar, "application/x-ustar" }, + { ContentType.Utz, "application/vnd.uiq.theme" }, + { ContentType.Uu, "text/x-uuencode" }, + { ContentType.Uva, "audio/vnd.dece.audio" }, + { ContentType.Uvd, "application/vnd.dece.data" }, + { ContentType.Uvf, "application/vnd.dece.data" }, + { ContentType.Uvg, "image/vnd.dece.graphic" }, + { ContentType.Uvh, "video/vnd.dece.hd" }, + { ContentType.Uvi, "image/vnd.dece.graphic" }, + { ContentType.Uvm, "video/vnd.dece.mobile" }, + { ContentType.Uvp, "video/vnd.dece.pd" }, + { ContentType.Uvs, "video/vnd.dece.sd" }, + { ContentType.Uvt, "application/vnd.dece.ttml+xml" }, + { ContentType.Uvu, "video/vnd.uvvu.mp4" }, + { ContentType.Uvv, "video/vnd.dece.video" }, + { ContentType.Uvva, "audio/vnd.dece.audio" }, + { ContentType.Uvvd, "application/vnd.dece.data" }, + { ContentType.Uvvf, "application/vnd.dece.data" }, + { ContentType.Uvvg, "image/vnd.dece.graphic" }, + { ContentType.Uvvh, "video/vnd.dece.hd" }, + { ContentType.Uvvi, "image/vnd.dece.graphic" }, + { ContentType.Uvvm, "video/vnd.dece.mobile" }, + { ContentType.Uvvp, "video/vnd.dece.pd" }, + { ContentType.Uvvs, "video/vnd.dece.sd" }, + { ContentType.Uvvt, "application/vnd.dece.ttml+xml" }, + { ContentType.Uvvu, "video/vnd.uvvu.mp4" }, + { ContentType.Uvvv, "video/vnd.dece.video" }, + { ContentType.Uvvx, "application/vnd.dece.unspecified" }, + { ContentType.Uvvz, "application/vnd.dece.zip" }, + { ContentType.Uvx, "application/vnd.dece.unspecified" }, + { ContentType.Uvz, "application/vnd.dece.zip" }, + { ContentType.Vbox, "application/x-virtualbox-vbox" }, + { ContentType.Vcard, "text/vcard" }, + { ContentType.Vcd, "application/x-cdlink" }, + { ContentType.Vcf, "text/x-vcard" }, + { ContentType.Vcg, "application/vnd.groove-vcard" }, + { ContentType.Vcs, "text/x-vcalendar" }, + { ContentType.Vcx, "application/vnd.vcx" }, + { ContentType.Vdi, "application/x-virtualbox-vdi" }, + { ContentType.Vds, "model/vnd.sap.vds" }, + { ContentType.Vhd, "application/x-virtualbox-vhd" }, + { ContentType.Vis, "application/vnd.visionary" }, + { ContentType.Viv, "video/vnd.vivo" }, + { ContentType.Vmdk, "application/x-virtualbox-vmdk" }, + { ContentType.Vob, "video/x-ms-vob" }, + { ContentType.Vor, "application/vnd.stardivision.writer" }, + { ContentType.Vox, "application/x-authorware-bin" }, + { ContentType.Vrml, "model/vrml" }, + { ContentType.Vsd, "application/vnd.visio" }, + { ContentType.Vsf, "application/vnd.vsf" }, + { ContentType.Vss, "application/vnd.visio" }, + { ContentType.Vst, "application/vnd.visio" }, + { ContentType.Vsw, "application/vnd.visio" }, + { ContentType.Vtf, "image/vnd.valve.source.texture" }, + { ContentType.Vtt, "text/vtt" }, + { ContentType.Vtu, "model/vnd.vtu" }, + { ContentType.Vxml, "application/voicexml+xml" }, + { ContentType.W3d, "application/x-director" }, + { ContentType.Wad, "application/x-doom" }, + { ContentType.Wadl, "application/vnd.sun.wadl+xml" }, + { ContentType.War, "application/java-archive" }, + { ContentType.Wasm, "application/wasm" }, + { ContentType.Wav, "audio/wav" }, + { ContentType.Wax, "audio/x-ms-wax" }, + { ContentType.Wbmp, "image/vnd.wap.wbmp" }, + { ContentType.Wbs, "application/vnd.criticaltools.wbs+xml" }, + { ContentType.Wbxml, "application/vnd.wap.wbxml" }, + { ContentType.Wcm, "application/vnd.ms-works" }, + { ContentType.Wdb, "application/vnd.ms-works" }, + { ContentType.Wdp, "image/vnd.ms-photo" }, + { ContentType.Weba, "audio/webm" }, + { ContentType.Webapp, "application/x-web-app-manifest+json" }, + { ContentType.Webm, "video/webm" }, + { ContentType.Webp, "image/webp" }, + { ContentType.Wg, "application/vnd.pmi.widget" }, + { ContentType.Wgt, "application/widget" }, + { ContentType.Wks, "application/vnd.ms-works" }, + { ContentType.Wm, "video/x-ms-wm" }, + { ContentType.Wma, "audio/x-ms-wma" }, + { ContentType.Wmd, "application/x-ms-wmd" }, + { ContentType.Wmf, "application/x-msmetafile" }, + { ContentType.Wml, "text/vnd.wap.wml" }, + { ContentType.Wmlc, "application/vnd.wap.wmlc" }, + { ContentType.Wmls, "text/vnd.wap.wmlscript" }, + { ContentType.Wmlsc, "application/vnd.wap.wmlscriptc" }, + { ContentType.Wmv, "video/x-ms-wmv" }, + { ContentType.Wmx, "video/x-ms-wmx" }, + { ContentType.Wmz, "application/x-ms-wmz" }, + { ContentType.Woff, "font/woff" }, + { ContentType.Woff2, "font/woff2" }, + { ContentType.Wpd, "application/vnd.wordperfect" }, + { ContentType.Wpl, "application/vnd.ms-wpl" }, + { ContentType.Wps, "application/vnd.ms-works" }, + { ContentType.Wqd, "application/vnd.wqd" }, + { ContentType.Wri, "application/x-mswrite" }, + { ContentType.Wrl, "model/vrml" }, + { ContentType.Wsc, "message/vnd.wfa.wsc" }, + { ContentType.Wsdl, "application/wsdl+xml" }, + { ContentType.Wspolicy, "application/wspolicy+xml" }, + { ContentType.Wtb, "application/vnd.webturbo" }, + { ContentType.Wvx, "video/x-ms-wvx" }, + { ContentType.X32, "application/x-authorware-bin" }, + { ContentType.X3d, "model/x3d+xml" }, + { ContentType.X3db, "model/x3d+binary" }, + { ContentType.X3dbz, "model/x3d+binary" }, + { ContentType.X3dv, "model/x3d+vrml" }, + { ContentType.X3dvz, "model/x3d+vrml" }, + { ContentType.X3dz, "model/x3d+xml" }, + { ContentType.Xaml, "application/xaml+xml" }, + { ContentType.Xap, "application/x-silverlight-app" }, + { ContentType.Xar, "application/vnd.xara" }, + { ContentType.Xav, "application/xcap-att+xml" }, + { ContentType.Xbap, "application/x-ms-xbap" }, + { ContentType.Xbd, "application/vnd.fujixerox.docuworks.binder" }, + { ContentType.Xbm, "image/x-xbitmap" }, + { ContentType.Xca, "application/xcap-caps+xml" }, + { ContentType.Xcs, "application/calendar+xml" }, + { ContentType.Xdf, "application/xcap-diff+xml" }, + { ContentType.Xdm, "application/vnd.syncml.dm+xml" }, + { ContentType.Xdp, "application/vnd.adobe.xdp+xml" }, + { ContentType.Xdssc, "application/dssc+xml" }, + { ContentType.Xdw, "application/vnd.fujixerox.docuworks" }, + { ContentType.Xel, "application/xcap-el+xml" }, + { ContentType.Xenc, "application/xenc+xml" }, + { ContentType.Xer, "application/patch-ops-error+xml" }, + { ContentType.Xfdf, "application/vnd.adobe.xfdf" }, + { ContentType.Xfdl, "application/vnd.xfdl" }, + { ContentType.Xht, "application/xhtml+xml" }, + { ContentType.Xhtml, "application/xhtml+xml" }, + { ContentType.Xhvml, "application/xv+xml" }, + { ContentType.Xif, "image/vnd.xiff" }, + { ContentType.Xla, "application/vnd.ms-excel" }, + { ContentType.Xlam, "application/vnd.ms-excel.addin.macroenabled.12" }, + { ContentType.Xlc, "application/vnd.ms-excel" }, + { ContentType.Xlf, "application/x-xliff+xml" }, + { ContentType.Xlm, "application/vnd.ms-excel" }, + { ContentType.Xls, "application/vnd.ms-excel" }, + { ContentType.Xlsb, "application/vnd.ms-excel.sheet.binary.macroenabled.12" }, + { ContentType.Xlsm, "application/vnd.ms-excel.sheet.macroenabled.12" }, + { ContentType.Xlsx, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" }, + { ContentType.Xlt, "application/vnd.ms-excel" }, + { ContentType.Xltm, "application/vnd.ms-excel.template.macroenabled.12" }, + { ContentType.Xltx, "application/vnd.openxmlformats-officedocument.spreadsheetml.template" }, + { ContentType.Xlw, "application/vnd.ms-excel" }, + { ContentType.Xm, "audio/xm" }, + { ContentType.Xml, "application/xml" }, + { ContentType.Xns, "application/xcap-ns+xml" }, + { ContentType.Xo, "application/vnd.olpc-sugar" }, + { ContentType.Xop, "application/xop+xml" }, + { ContentType.Xpi, "application/x-xpinstall" }, + { ContentType.Xpl, "application/xproc+xml" }, + { ContentType.Xpm, "image/x-xpixmap" }, + { ContentType.Xpr, "application/vnd.is-xpr" }, + { ContentType.Xps, "application/vnd.ms-xpsdocument" }, + { ContentType.Xpw, "application/vnd.intercon.formnet" }, + { ContentType.Xpx, "application/vnd.intercon.formnet" }, + { ContentType.Xsd, "application/xml" }, + { ContentType.Xsl, "application/xml" }, + { ContentType.Xslt, "application/xslt+xml" }, + { ContentType.Xsm, "application/vnd.syncml+xml" }, + { ContentType.Xspf, "application/xspf+xml" }, + { ContentType.Xul, "application/vnd.mozilla.xul+xml" }, + { ContentType.Xvm, "application/xv+xml" }, + { ContentType.Xvml, "application/xv+xml" }, + { ContentType.Xwd, "image/x-xwindowdump" }, + { ContentType.Xyz, "chemical/x-xyz" }, + { ContentType.Xz, "application/x-xz" }, + { ContentType.Yaml, "text/yaml" }, + { ContentType.Yang, "application/yang" }, + { ContentType.Yin, "application/yin+xml" }, + { ContentType.Yml, "text/yaml" }, + { ContentType.Ymp, "text/x-suse-ymp" }, + { ContentType.Z1, "application/x-zmachine" }, + { ContentType.Z2, "application/x-zmachine" }, + { ContentType.Z3, "application/x-zmachine" }, + { ContentType.Z4, "application/x-zmachine" }, + { ContentType.Z5, "application/x-zmachine" }, + { ContentType.Z6, "application/x-zmachine" }, + { ContentType.Z7, "application/x-zmachine" }, + { ContentType.Z8, "application/x-zmachine" }, + { ContentType.Zaz, "application/vnd.zzazz.deck+xml" }, + { ContentType.Zip, "application/zip" }, + { ContentType.Zir, "application/vnd.zul" }, + { ContentType.Zirz, "application/vnd.zul" }, + { ContentType.Zmm, "application/vnd.handheld-entertainment+xml" }, + }; + private static readonly IReadOnlyDictionary<string, ContentType> ExtensionToCt = new Dictionary<string, ContentType>() + { + { "aab", ContentType.Aab }, + { "aac", ContentType.Aac }, + { "aam", ContentType.Aam }, + { "aas", ContentType.Aas }, + { "abw", ContentType.Abw }, + { "ac", ContentType.Ac }, + { "acc", ContentType.Acc }, + { "ace", ContentType.Ace }, + { "acu", ContentType.Acu }, + { "acutc", ContentType.Acutc }, + { "adp", ContentType.Adp }, + { "aep", ContentType.Aep }, + { "afm", ContentType.Afm }, + { "afp", ContentType.Afp }, + { "ahead", ContentType.Ahead }, + { "ai", ContentType.Ai }, + { "aif", ContentType.Aif }, + { "aifc", ContentType.Aifc }, + { "aiff", ContentType.Aiff }, + { "air", ContentType.Air }, + { "ait", ContentType.Ait }, + { "ami", ContentType.Ami }, + { "amr", ContentType.Amr }, + { "apk", ContentType.Apk }, + { "apng", ContentType.Apng }, + { "appcache", ContentType.Appcache }, + { "apr", ContentType.Apr }, + { "arc", ContentType.Arc }, + { "arj", ContentType.Arj }, + { "asc", ContentType.Asc }, + { "asf", ContentType.Asf }, + { "asm", ContentType.Asm }, + { "aso", ContentType.Aso }, + { "asx", ContentType.Asx }, + { "atc", ContentType.Atc }, + { "atom", ContentType.Atom }, + { "atomcat", ContentType.Atomcat }, + { "atomsvc", ContentType.Atomsvc }, + { "atx", ContentType.Atx }, + { "au", ContentType.Au }, + { "avi", ContentType.Avi }, + { "avif", ContentType.Avif }, + { "aw", ContentType.Aw }, + { "azf", ContentType.Azf }, + { "azs", ContentType.Azs }, + { "azv", ContentType.Azv }, + { "azw", ContentType.Azw }, + { "b16", ContentType.B16 }, + { "bat", ContentType.Bat }, + { "bcpio", ContentType.Bcpio }, + { "bdf", ContentType.Bdf }, + { "bdm", ContentType.Bdm }, + { "bdoc", ContentType.Bdoc }, + { "bed", ContentType.Bed }, + { "bh2", ContentType.Bh2 }, + { "bin", ContentType.Binary }, + { "blb", ContentType.Blb }, + { "blorb", ContentType.Blorb }, + { "bmi", ContentType.Bmi }, + { "bmml", ContentType.Bmml }, + { "bmp", ContentType.Bmp }, + { "book", ContentType.Book }, + { "box", ContentType.Box }, + { "boz", ContentType.Boz }, + { "bpk", ContentType.Bpk }, + { "bsp", ContentType.Bsp }, + { "btif", ContentType.Btif }, + { "buffer", ContentType.Buffer }, + { "bz", ContentType.Bz }, + { "bz2", ContentType.Bz2 }, + { "c", ContentType.C }, + { "c11amc", ContentType.C11amc }, + { "c11amz", ContentType.C11amz }, + { "c4d", ContentType.C4d }, + { "c4f", ContentType.C4f }, + { "c4g", ContentType.C4g }, + { "c4p", ContentType.C4p }, + { "c4u", ContentType.C4u }, + { "cab", ContentType.Cab }, + { "caf", ContentType.Caf }, + { "cap", ContentType.Cap }, + { "car", ContentType.Car }, + { "cat", ContentType.Cat }, + { "cb7", ContentType.Cb7 }, + { "cba", ContentType.Cba }, + { "cbr", ContentType.Cbr }, + { "cbt", ContentType.Cbt }, + { "cbz", ContentType.Cbz }, + { "cc", ContentType.Cc }, + { "cco", ContentType.Cco }, + { "cct", ContentType.Cct }, + { "ccxml", ContentType.Ccxml }, + { "cdbcmsg", ContentType.Cdbcmsg }, + { "cdf", ContentType.Cdf }, + { "cdfx", ContentType.Cdfx }, + { "cdkey", ContentType.Cdkey }, + { "cdmia", ContentType.Cdmia }, + { "cdmic", ContentType.Cdmic }, + { "cdmid", ContentType.Cdmid }, + { "cdmio", ContentType.Cdmio }, + { "cdmiq", ContentType.Cdmiq }, + { "cdx", ContentType.Cdx }, + { "cdxml", ContentType.Cdxml }, + { "cdy", ContentType.Cdy }, + { "cer", ContentType.Cer }, + { "cfs", ContentType.Cfs }, + { "cgm", ContentType.Cgm }, + { "chat", ContentType.Chat }, + { "chm", ContentType.Chm }, + { "chrt", ContentType.Chrt }, + { "cif", ContentType.Cif }, + { "cii", ContentType.Cii }, + { "cil", ContentType.Cil }, + { "cjs", ContentType.Cjs }, + { "cla", ContentType.Cla }, + { "clkk", ContentType.Clkk }, + { "clkp", ContentType.Clkp }, + { "clkt", ContentType.Clkt }, + { "clkw", ContentType.Clkw }, + { "clkx", ContentType.Clkx }, + { "clp", ContentType.Clp }, + { "cmc", ContentType.Cmc }, + { "cmdf", ContentType.Cmdf }, + { "cml", ContentType.Cml }, + { "cmp", ContentType.Cmp }, + { "cmx", ContentType.Cmx }, + { "cod", ContentType.Cod }, + { "coffee", ContentType.Coffee }, + { "com", ContentType.Com }, + { "conf", ContentType.Conf }, + { "cpio", ContentType.Cpio }, + { "cpp", ContentType.Cpp }, + { "cpt", ContentType.Cpt }, + { "crd", ContentType.Crd }, + { "crl", ContentType.Crl }, + { "crt", ContentType.Crt }, + { "crx", ContentType.Crx }, + { "csh", ContentType.Csh }, + { "csl", ContentType.Csl }, + { "csml", ContentType.Csml }, + { "csp", ContentType.Csp }, + { "css", ContentType.Css }, + { "cst", ContentType.Cst }, + { "csv", ContentType.Csv }, + { "cu", ContentType.Cu }, + { "curl", ContentType.Curl }, + { "cww", ContentType.Cww }, + { "cxt", ContentType.Cxt }, + { "cxx", ContentType.Cxx }, + { "dae", ContentType.Dae }, + { "daf", ContentType.Daf }, + { "dart", ContentType.Dart }, + { "dataless", ContentType.Dataless }, + { "davmount", ContentType.Davmount }, + { "dbf", ContentType.Dbf }, + { "dbk", ContentType.Dbk }, + { "dcr", ContentType.Dcr }, + { "dcurl", ContentType.Dcurl }, + { "dd2", ContentType.Dd2 }, + { "ddd", ContentType.Ddd }, + { "ddf", ContentType.Ddf }, + { "dds", ContentType.Dds }, + { "deb", ContentType.Deb }, + { "def", ContentType.Def }, + { "deploy", ContentType.Deploy }, + { "der", ContentType.Der }, + { "dfac", ContentType.Dfac }, + { "dgc", ContentType.Dgc }, + { "dic", ContentType.Dic }, + { "dir", ContentType.Dir }, + { "dis", ContentType.Dis }, + { "dist", ContentType.Dist }, + { "distz", ContentType.Distz }, + { "djv", ContentType.Djv }, + { "djvu", ContentType.Djvu }, + { "dll", ContentType.Dll }, + { "dmg", ContentType.Dmg }, + { "dmp", ContentType.Dmp }, + { "dms", ContentType.Dms }, + { "dna", ContentType.Dna }, + { "doc", ContentType.Doc }, + { "docm", ContentType.Docm }, + { "docx", ContentType.Docx }, + { "dot", ContentType.Dot }, + { "dotm", ContentType.Dotm }, + { "dotx", ContentType.Dotx }, + { "dp", ContentType.Dp }, + { "dpg", ContentType.Dpg }, + { "dra", ContentType.Dra }, + { "drle", ContentType.Drle }, + { "dsc", ContentType.Dsc }, + { "dssc", ContentType.Dssc }, + { "dtb", ContentType.Dtb }, + { "dtd", ContentType.Dtd }, + { "dts", ContentType.Dts }, + { "dtshd", ContentType.Dtshd }, + { "dump", ContentType.Dump }, + { "dvb", ContentType.Dvb }, + { "dvi", ContentType.Dvi }, + { "dwd", ContentType.Dwd }, + { "dwf", ContentType.Dwf }, + { "dwg", ContentType.Dwg }, + { "dxf", ContentType.Dxf }, + { "dxp", ContentType.Dxp }, + { "dxr", ContentType.Dxr }, + { "ear", ContentType.Ear }, + { "ecma", ContentType.Ecma }, + { "edm", ContentType.Edm }, + { "edx", ContentType.Edx }, + { "efif", ContentType.Efif }, + { "ei6", ContentType.Ei6 }, + { "elc", ContentType.Elc }, + { "emf", ContentType.Emf }, + { "eml", ContentType.Eml }, + { "emma", ContentType.Emma }, + { "emz", ContentType.Emz }, + { "eol", ContentType.Eol }, + { "eot", ContentType.Eot }, + { "eps", ContentType.Eps }, + { "epub", ContentType.Epub }, + { "es", ContentType.Es }, + { "es3", ContentType.Es3 }, + { "esa", ContentType.Esa }, + { "esf", ContentType.Esf }, + { "et3", ContentType.Et3 }, + { "etx", ContentType.Etx }, + { "eva", ContentType.Eva }, + { "evy", ContentType.Evy }, + { "exe", ContentType.Exe }, + { "exi", ContentType.Exi }, + { "exp", ContentType.Exp }, + { "exr", ContentType.Exr }, + { "ext", ContentType.Ext }, + { "ez", ContentType.Ez }, + { "ez2", ContentType.Ez2 }, + { "ez3", ContentType.Ez3 }, + { "f", ContentType.F }, + { "f4v", ContentType.F4v }, + { "f77", ContentType.Fortran }, + { "f90", ContentType.F90 }, + { "fbs", ContentType.Fbs }, + { "fcdt", ContentType.Fcdt }, + { "fcs", ContentType.Fcs }, + { "fdf", ContentType.Fdf }, + { "fdt", ContentType.Fdt }, + { "fg5", ContentType.Fg5 }, + { "fgd", ContentType.Fgd }, + { "fh", ContentType.Fh }, + { "fh4", ContentType.Fh4 }, + { "fh5", ContentType.Fh5 }, + { "fh7", ContentType.Fh7 }, + { "fhc", ContentType.Fhc }, + { "fig", ContentType.Fig }, + { "fits", ContentType.Fits }, + { "flac", ContentType.Flac }, + { "fli", ContentType.Fli }, + { "flo", ContentType.Flo }, + { "flv", ContentType.Flv }, + { "flw", ContentType.Flw }, + { "flx", ContentType.Flx }, + { "fly", ContentType.Fly }, + { "fm", ContentType.Fm }, + { "fnc", ContentType.Fnc }, + { "fo", ContentType.Fo }, + { "for", ContentType.For }, + { "fpx", ContentType.Fpx }, + { "frame", ContentType.Frame }, + { "fsc", ContentType.Fsc }, + { "fst", ContentType.Fst }, + { "ftc", ContentType.Ftc }, + { "fti", ContentType.Fti }, + { "fvt", ContentType.Fvt }, + { "fxp", ContentType.Fxp }, + { "fxpl", ContentType.Fxpl }, + { "fzs", ContentType.Fzs }, + { "g2w", ContentType.G2w }, + { "g3", ContentType.G3 }, + { "g3w", ContentType.G3w }, + { "gac", ContentType.Gac }, + { "gam", ContentType.Gam }, + { "gbr", ContentType.Gbr }, + { "gca", ContentType.Gca }, + { "gdl", ContentType.Gdl }, + { "gdoc", ContentType.Gdoc }, + { "geo", ContentType.Geo }, + { "geojson", ContentType.Geojson }, + { "gex", ContentType.Gex }, + { "ggb", ContentType.Ggb }, + { "ggt", ContentType.Ggt }, + { "ghf", ContentType.Ghf }, + { "gif", ContentType.Gif }, + { "gim", ContentType.Gim }, + { "glb", ContentType.Glb }, + { "gltf", ContentType.Gltf }, + { "gml", ContentType.Gml }, + { "gmx", ContentType.Gmx }, + { "gnumeric", ContentType.Gnumeric }, + { "gph", ContentType.Gph }, + { "gpx", ContentType.Gpx }, + { "gqf", ContentType.Gqf }, + { "gqs", ContentType.Gqs }, + { "gram", ContentType.Gram }, + { "gramps", ContentType.Gramps }, + { "gre", ContentType.Gre }, + { "grv", ContentType.Grv }, + { "grxml", ContentType.Grxml }, + { "gsf", ContentType.Gsf }, + { "gsheet", ContentType.Gsheet }, + { "gslides", ContentType.Gslides }, + { "gtar", ContentType.Gtar }, + { "gtm", ContentType.Gtm }, + { "gtw", ContentType.Gtw }, + { "gv", ContentType.Gv }, + { "gxf", ContentType.Gxf }, + { "gxt", ContentType.Gxt }, + { "gz", ContentType.Gz }, + { "h", ContentType.H }, + { "h261", ContentType.H261 }, + { "h263", ContentType.H263 }, + { "h264", ContentType.H264 }, + { "hal", ContentType.Hal }, + { "hbci", ContentType.Hbci }, + { "hbs", ContentType.Hbs }, + { "hdd", ContentType.Hdd }, + { "hdf", ContentType.Hdf }, + { "heic", ContentType.Heic }, + { "heics", ContentType.Heics }, + { "heif", ContentType.Heif }, + { "heifs", ContentType.Heifs }, + { "hej2", ContentType.Hej2 }, + { "held", ContentType.Held }, + { "hh", ContentType.Hh }, + { "hjson", ContentType.Hjson }, + { "hlp", ContentType.Hlp }, + { "hpgl", ContentType.Hpgl }, + { "hpid", ContentType.Hpid }, + { "hps", ContentType.Hps }, + { "hqx", ContentType.Hqx }, + { "hsj2", ContentType.Hsj2 }, + { "htc", ContentType.Htc }, + { "htke", ContentType.Htke }, + { "htm", ContentType.Htm }, + { "html", ContentType.Html }, + { "hvd", ContentType.Hvd }, + { "hvp", ContentType.Hvp }, + { "hex", ContentType.Binary }, + { "hvs", ContentType.Hvs }, + { "i2g", ContentType.I2g }, + { "icc", ContentType.Icc }, + { "ice", ContentType.Ice }, + { "icm", ContentType.Icm }, + { "ico", ContentType.Ico }, + { "ics", ContentType.Ics }, + { "ief", ContentType.Ief }, + { "ifb", ContentType.Ifb }, + { "ifm", ContentType.Ifm }, + { "iges", ContentType.Iges }, + { "igl", ContentType.Igl }, + { "igm", ContentType.Igm }, + { "igs", ContentType.Igs }, + { "igx", ContentType.Igx }, + { "iif", ContentType.Iif }, + { "img", ContentType.Img }, + { "imp", ContentType.Imp }, + { "ims", ContentType.Ims }, + { "ini", ContentType.Ini }, + { "ink", ContentType.Ink }, + { "inkml", ContentType.Inkml }, + { "install", ContentType.Install }, + { "iota", ContentType.Iota }, + { "ipfix", ContentType.Ipfix }, + { "ipk", ContentType.Ipk }, + { "irm", ContentType.Irm }, + { "irp", ContentType.Irp }, + { "iso", ContentType.Iso }, + { "itp", ContentType.Itp }, + { "its", ContentType.Its }, + { "ivp", ContentType.Ivp }, + { "ivu", ContentType.Ivu }, + { "jad", ContentType.Jad }, + { "jade", ContentType.Jade }, + { "jam", ContentType.Jam }, + { "jar", ContentType.Jar }, + { "jardiff", ContentType.Jardiff }, + { "java", ContentType.Java }, + { "jhc", ContentType.Jhc }, + { "jisp", ContentType.Jisp }, + { "jls", ContentType.Jls }, + { "jlt", ContentType.Jlt }, + { "jng", ContentType.Jng }, + { "jnlp", ContentType.Jnlp }, + { "joda", ContentType.Joda }, + { "jp2", ContentType.Jp2 }, + { "jpe", ContentType.Jpe }, + { "jpeg", ContentType.Jpeg }, + { "jpf", ContentType.Jpf }, + { "jpg", ContentType.Jpg }, + { "jpg2", ContentType.Jpg2 }, + { "jpgm", ContentType.Jpgm }, + { "jpgv", ContentType.Jpgv }, + { "jph", ContentType.Jph }, + { "jpm", ContentType.Jpm }, + { "jpx", ContentType.Jpx }, + { "js", ContentType.Javascript }, + { "json", ContentType.Json }, + { "json5", ContentType.Json5 }, + { "jsonld", ContentType.Jsonld }, + { "jsonml", ContentType.Jsonml }, + { "jsx", ContentType.Jsx }, + { "jxr", ContentType.Jxr }, + { "jxra", ContentType.Jxra }, + { "jxrs", ContentType.Jxrs }, + { "jxs", ContentType.Jxs }, + { "jxsc", ContentType.Jxsc }, + { "jxsi", ContentType.Jxsi }, + { "jxss", ContentType.Jxss }, + { "kar", ContentType.Kar }, + { "karbon", ContentType.Karbon }, + { "kdbx", ContentType.Kdbx }, + { "key", ContentType.Key }, + { "kfo", ContentType.Kfo }, + { "kia", ContentType.Kia }, + { "kml", ContentType.Kml }, + { "kmz", ContentType.Kmz }, + { "kne", ContentType.Kne }, + { "knp", ContentType.Knp }, + { "kon", ContentType.Kon }, + { "kpr", ContentType.Kpr }, + { "kpt", ContentType.Kpt }, + { "kpxx", ContentType.Kpxx }, + { "ksp", ContentType.Ksp }, + { "ktr", ContentType.Ktr }, + { "ktx", ContentType.Ktx }, + { "ktx2", ContentType.Ktx2 }, + { "ktz", ContentType.Ktz }, + { "kwd", ContentType.Kwd }, + { "kwt", ContentType.Kwt }, + { "lasxml", ContentType.Lasxml }, + { "latex", ContentType.Latex }, + { "lbd", ContentType.Lbd }, + { "lbe", ContentType.Lbe }, + { "les", ContentType.Les }, + { "less", ContentType.Less }, + { "lgr", ContentType.Lgr }, + { "lha", ContentType.Lha }, + { "link66", ContentType.Link66 }, + { "list", ContentType.List }, + { "list3820", ContentType.List3820 }, + { "listafp", ContentType.Listafp }, + { "lnk", ContentType.Lnk }, + { "log", ContentType.Log }, + { "lostxml", ContentType.Lostxml }, + { "lrf", ContentType.Lrf }, + { "lrm", ContentType.Lrm }, + { "ltf", ContentType.Ltf }, + { "lua", ContentType.Lua }, + { "luac", ContentType.Luac }, + { "lvp", ContentType.Lvp }, + { "lwp", ContentType.Lwp }, + { "lzh", ContentType.Lzh }, + { "m13", ContentType.M13 }, + { "m14", ContentType.M14 }, + { "m1v", ContentType.M1v }, + { "m21", ContentType.M21 }, + { "m2a", ContentType.M2a }, + { "m2v", ContentType.M2v }, + { "m3a", ContentType.M3a }, + { "m3u", ContentType.M3u }, + { "m3u8", ContentType.M3u8 }, + { "m4a", ContentType.M4a }, + { "m4p", ContentType.M4p }, + { "m4s", ContentType.M4s }, + { "m4u", ContentType.M4u }, + { "m4v", ContentType.M4v }, + { "ma", ContentType.Ma }, + { "mads", ContentType.Mads }, + { "maei", ContentType.Maei }, + { "mag", ContentType.Mag }, + { "maker", ContentType.Maker }, + { "man", ContentType.Man }, + { "manifest", ContentType.Manifest }, + { "map", ContentType.Map }, + { "mar", ContentType.Mar }, + { "markdown", ContentType.Markdown }, + { "mathml", ContentType.Mathml }, + { "mb", ContentType.Mb }, + { "mbk", ContentType.Mbk }, + { "mbox", ContentType.Mbox }, + { "mc1", ContentType.Mc1 }, + { "mcd", ContentType.Mcd }, + { "mcurl", ContentType.Mcurl }, + { "md", ContentType.Md }, + { "mdb", ContentType.Mdb }, + { "mdi", ContentType.Mdi }, + { "mdx", ContentType.Mdx }, + { "me", ContentType.Me }, + { "mesh", ContentType.Mesh }, + { "meta4", ContentType.Meta4 }, + { "metalink", ContentType.Metalink }, + { "mets", ContentType.Mets }, + { "mfm", ContentType.Mfm }, + { "mft", ContentType.Mft }, + { "mgp", ContentType.Mgp }, + { "mgz", ContentType.Mgz }, + { "mid", ContentType.Mid }, + { "midi", ContentType.Midi }, + { "mie", ContentType.Mie }, + { "mif", ContentType.Mif }, + { "mime", ContentType.Mime }, + { "mj2", ContentType.Mj2 }, + { "mjp2", ContentType.Mjp2 }, + { "mjs", ContentType.Mjs }, + { "mk3d", ContentType.Mk3d }, + { "mka", ContentType.Mka }, + { "mkd", ContentType.Mkd }, + { "mks", ContentType.Mks }, + { "mkv", ContentType.Mkv }, + { "mlp", ContentType.Mlp }, + { "mmd", ContentType.Mmd }, + { "mmf", ContentType.Mmf }, + { "mml", ContentType.Mml }, + { "mmr", ContentType.Mmr }, + { "mng", ContentType.Mng }, + { "mny", ContentType.Mny }, + { "mobi", ContentType.Mobi }, + { "mods", ContentType.Mods }, + { "mov", ContentType.Mov }, + { "movie", ContentType.Movie }, + { "mp2", ContentType.Mp2 }, + { "mp21", ContentType.Mp21 }, + { "mp2a", ContentType.Mp2a }, + { "mp3", ContentType.Mp3 }, + { "mp4", ContentType.Mp4 }, + { "mp4a", ContentType.Mp4a }, + { "mp4s", ContentType.Mp4s }, + { "mp4v", ContentType.Mp4v }, + { "mpc", ContentType.Mpc }, + { "mpd", ContentType.Mpd }, + { "mpe", ContentType.Mpe }, + { "mpeg", ContentType.Mpeg }, + { "mpg", ContentType.Mpg }, + { "mpg4", ContentType.Mpg4 }, + { "mpga", ContentType.Mpga }, + { "mpkg", ContentType.Mpkg }, + { "mpm", ContentType.Mpm }, + { "mpn", ContentType.Mpn }, + { "mpp", ContentType.Mpp }, + { "mpt", ContentType.Mpt }, + { "mpy", ContentType.Mpy }, + { "mqy", ContentType.Mqy }, + { "mrc", ContentType.Mrc }, + { "mrcx", ContentType.Mrcx }, + { "ms", ContentType.Ms }, + { "mscml", ContentType.Mscml }, + { "mseed", ContentType.Mseed }, + { "mseq", ContentType.Mseq }, + { "msf", ContentType.Msf }, + { "msg", ContentType.Msg }, + { "msh", ContentType.Msh }, + { "msi", ContentType.Msi }, + { "msl", ContentType.Msl }, + { "msm", ContentType.Msm }, + { "msp", ContentType.Msp }, + { "msty", ContentType.Msty }, + { "mtl", ContentType.Mtl }, + { "mts", ContentType.Mts }, + { "mus", ContentType.Mus }, + { "musd", ContentType.Musd }, + { "musicxml", ContentType.Musicxml }, + { "mvb", ContentType.Mvb }, + { "mvt", ContentType.Mvt }, + { "mwf", ContentType.Mwf }, + { "mxf", ContentType.Mxf }, + { "mxl", ContentType.Mxl }, + { "mxmf", ContentType.Mxmf }, + { "mxml", ContentType.Mxml }, + { "mxs", ContentType.Mxs }, + { "mxu", ContentType.Mxu }, + { "n3", ContentType.N3 }, + { "nb", ContentType.Nb }, + { "nbp", ContentType.Nbp }, + { "nc", ContentType.Nc }, + { "ncx", ContentType.Ncx }, + { "nfo", ContentType.Nfo }, + { "ngdat", ContentType.Ngdat }, + { "nitf", ContentType.Nitf }, + { "nlu", ContentType.Nlu }, + { "nml", ContentType.Nml }, + { "nnd", ContentType.Nnd }, + { "nns", ContentType.Nns }, + { "nnw", ContentType.Nnw }, + { "npx", ContentType.Npx }, + { "nq", ContentType.Nq }, + { "nsc", ContentType.Nsc }, + { "nsf", ContentType.Nsf }, + { "nt", ContentType.Nt }, + { "ntf", ContentType.Ntf }, + { "numbers", ContentType.Numbers }, + { "nzb", ContentType.Nzb }, + { "oa2", ContentType.Oa2 }, + { "oa3", ContentType.Oa3 }, + { "oas", ContentType.Oas }, + { "obd", ContentType.Obd }, + { "obgx", ContentType.Obgx }, + { "obj", ContentType.Obj }, + { "oda", ContentType.Oda }, + { "odb", ContentType.Odb }, + { "odc", ContentType.Odc }, + { "odf", ContentType.Odf }, + { "odft", ContentType.Odft }, + { "odg", ContentType.Odg }, + { "odi", ContentType.Odi }, + { "odm", ContentType.Odm }, + { "odp", ContentType.Odp }, + { "ods", ContentType.Ods }, + { "odt", ContentType.Odt }, + { "oga", ContentType.Oga }, + { "ogex", ContentType.Ogex }, + { "ogg", ContentType.Ogg }, + { "ogv", ContentType.Ogv }, + { "ogx", ContentType.Ogx }, + { "omdoc", ContentType.Omdoc }, + { "onepkg", ContentType.Onepkg }, + { "onetmp", ContentType.Onetmp }, + { "onetoc", ContentType.Onetoc }, + { "onetoc2", ContentType.Onetoc2 }, + { "opf", ContentType.Opf }, + { "opml", ContentType.Opml }, + { "oprc", ContentType.Oprc }, + { "opus", ContentType.Opus }, + { "org", ContentType.Org }, + { "osf", ContentType.Osf }, + { "osfpvg", ContentType.Osfpvg }, + { "osm", ContentType.Osm }, + { "otc", ContentType.Otc }, + { "otf", ContentType.Otf }, + { "otg", ContentType.Otg }, + { "oth", ContentType.Oth }, + { "oti", ContentType.Oti }, + { "otp", ContentType.Otp }, + { "ots", ContentType.Ots }, + { "ott", ContentType.Ott }, + { "ova", ContentType.Ova }, + { "ovf", ContentType.Ovf }, + { "owl", ContentType.Owl }, + { "oxps", ContentType.Oxps }, + { "oxt", ContentType.Oxt }, + { "p", ContentType.P }, + { "p10", ContentType.P10 }, + { "p12", ContentType.P12 }, + { "p7b", ContentType.P7b }, + { "p7c", ContentType.P7c }, + { "p7m", ContentType.P7m }, + { "p7r", ContentType.P7r }, + { "p7s", ContentType.P7s }, + { "p8", ContentType.P8 }, + { "pac", ContentType.Pac }, + { "pages", ContentType.Pages }, + { "pas", ContentType.Pas }, + { "paw", ContentType.Paw }, + { "pbd", ContentType.Pbd }, + { "pbm", ContentType.Pbm }, + { "pcap", ContentType.Pcap }, + { "pcf", ContentType.Pcf }, + { "pcl", ContentType.Pcl }, + { "pclxl", ContentType.Pclxl }, + { "pct", ContentType.Pct }, + { "pcurl", ContentType.Pcurl }, + { "pcx", ContentType.Pcx }, + { "pdb", ContentType.Pdb }, + { "pde", ContentType.Pde }, + { "pdf", ContentType.Pdf }, + { "pem", ContentType.Pem }, + { "pfa", ContentType.Pfa }, + { "pfb", ContentType.Pfb }, + { "pfm", ContentType.Pfm }, + { "pfr", ContentType.Pfr }, + { "pfx", ContentType.Pfx }, + { "pgm", ContentType.Pgm }, + { "pgn", ContentType.Pgn }, + { "pgp", ContentType.Pgp }, + { "php", ContentType.Php }, + { "pic", ContentType.Pic }, + { "pkg", ContentType.Pkg }, + { "pki", ContentType.Pki }, + { "pkipath", ContentType.Pkipath }, + { "pkpass", ContentType.Pkpass }, + { "pl", ContentType.Pl }, + { "plb", ContentType.Plb }, + { "plc", ContentType.Plc }, + { "plf", ContentType.Plf }, + { "pls", ContentType.Pls }, + { "pm", ContentType.Pm }, + { "pml", ContentType.Pml }, + { "png", ContentType.Png }, + { "pnm", ContentType.Pnm }, + { "portpkg", ContentType.Portpkg }, + { "pot", ContentType.Pot }, + { "potm", ContentType.Potm }, + { "potx", ContentType.Potx }, + { "ppam", ContentType.Ppam }, + { "ppd", ContentType.Ppd }, + { "ppm", ContentType.Ppm }, + { "pps", ContentType.Pps }, + { "ppsm", ContentType.Ppsm }, + { "ppsx", ContentType.Ppsx }, + { "ppt", ContentType.Ppt }, + { "pptm", ContentType.Pptm }, + { "pptx", ContentType.Pptx }, + { "pqa", ContentType.Pqa }, + { "prc", ContentType.Prc }, + { "pre", ContentType.Pre }, + { "prf", ContentType.Prf }, + { "provx", ContentType.Provx }, + { "ps", ContentType.Ps }, + { "psb", ContentType.Psb }, + { "psd", ContentType.Psd }, + { "psf", ContentType.Psf }, + { "pskcxml", ContentType.Pskcxml }, + { "pti", ContentType.Pti }, + { "ptid", ContentType.Ptid }, + { "pub", ContentType.Pub }, + { "pvb", ContentType.Pvb }, + { "pwn", ContentType.Pwn }, + { "pya", ContentType.Pya }, + { "pyv", ContentType.Pyv }, + { "qam", ContentType.Qam }, + { "qbo", ContentType.Qbo }, + { "qfx", ContentType.Qfx }, + { "qps", ContentType.Qps }, + { "qt", ContentType.Qt }, + { "qwd", ContentType.Qwd }, + { "qwt", ContentType.Qwt }, + { "qxb", ContentType.Qxb }, + { "qxd", ContentType.Qxd }, + { "qxl", ContentType.Qxl }, + { "qxt", ContentType.Qxt }, + { "ra", ContentType.Ra }, + { "ram", ContentType.Ram }, + { "raml", ContentType.Raml }, + { "rapd", ContentType.Rapd }, + { "rar", ContentType.Rar }, + { "ras", ContentType.Ras }, + { "rdf", ContentType.Rdf }, + { "rdz", ContentType.Rdz }, + { "relo", ContentType.Relo }, + { "rep", ContentType.Rep }, + { "res", ContentType.Res }, + { "rgb", ContentType.Rgb }, + { "rif", ContentType.Rif }, + { "rip", ContentType.Rip }, + { "ris", ContentType.Ris }, + { "rl", ContentType.Rl }, + { "rlc", ContentType.Rlc }, + { "rld", ContentType.Rld }, + { "rm", ContentType.Rm }, + { "rmi", ContentType.Rmi }, + { "rmp", ContentType.Rmp }, + { "rms", ContentType.Rms }, + { "rmvb", ContentType.Rmvb }, + { "rnc", ContentType.Rnc }, + { "rng", ContentType.Rng }, + { "roa", ContentType.Roa }, + { "roff", ContentType.Roff }, + { "rp9", ContentType.Rp9 }, + { "rpm", ContentType.Rpm }, + { "rpss", ContentType.Rpss }, + { "rpst", ContentType.Rpst }, + { "rq", ContentType.Rq }, + { "rs", ContentType.Rs }, + { "rsat", ContentType.Rsat }, + { "rsd", ContentType.Rsd }, + { "rsheet", ContentType.Rsheet }, + { "rss", ContentType.Rss }, + { "rtf", ContentType.Rtf }, + { "rtx", ContentType.Rtx }, + { "run", ContentType.Run }, + { "rusd", ContentType.Rusd }, + { "s", ContentType.S }, + { "s3m", ContentType.S3m }, + { "saf", ContentType.Saf }, + { "sass", ContentType.Sass }, + { "sbml", ContentType.Sbml }, + { "sc", ContentType.Sc }, + { "scd", ContentType.Scd }, + { "scm", ContentType.Scm }, + { "scq", ContentType.Scq }, + { "scs", ContentType.Scs }, + { "scss", ContentType.Scss }, + { "scurl", ContentType.Scurl }, + { "sda", ContentType.Sda }, + { "sdc", ContentType.Sdc }, + { "sdd", ContentType.Sdd }, + { "sdkd", ContentType.Sdkd }, + { "sdkm", ContentType.Sdkm }, + { "sdp", ContentType.Sdp }, + { "sdw", ContentType.Sdw }, + { "sea", ContentType.Sea }, + { "see", ContentType.See }, + { "seed", ContentType.Seed }, + { "sema", ContentType.Sema }, + { "semd", ContentType.Semd }, + { "semf", ContentType.Semf }, + { "senmlx", ContentType.Senmlx }, + { "sensmlx", ContentType.Sensmlx }, + { "ser", ContentType.Ser }, + { "setpay", ContentType.Setpay }, + { "setreg", ContentType.Setreg }, + { "sfs", ContentType.Sfs }, + { "sfv", ContentType.Sfv }, + { "sgi", ContentType.Sgi }, + { "sgl", ContentType.Sgl }, + { "sgm", ContentType.Sgm }, + { "sgml", ContentType.Sgml }, + { "sh", ContentType.Sh }, + { "shar", ContentType.Shar }, + { "shex", ContentType.Shex }, + { "shf", ContentType.Shf }, + { "shtml", ContentType.Shtml }, + { "sid", ContentType.Sid }, + { "sieve", ContentType.Sieve }, + { "sig", ContentType.Sig }, + { "sil", ContentType.Sil }, + { "silo", ContentType.Silo }, + { "sis", ContentType.Sis }, + { "sisx", ContentType.Sisx }, + { "sit", ContentType.Sit }, + { "sitx", ContentType.Sitx }, + { "siv", ContentType.Siv }, + { "skd", ContentType.Skd }, + { "skm", ContentType.Skm }, + { "skp", ContentType.Skp }, + { "skt", ContentType.Skt }, + { "sldm", ContentType.Sldm }, + { "sldx", ContentType.Sldx }, + { "slim", ContentType.Slim }, + { "slm", ContentType.Slm }, + { "sls", ContentType.Sls }, + { "slt", ContentType.Slt }, + { "sm", ContentType.Sm }, + { "smf", ContentType.Smf }, + { "smi", ContentType.Smi }, + { "smil", ContentType.Smil }, + { "smv", ContentType.Smv }, + { "smzip", ContentType.Smzip }, + { "snd", ContentType.Snd }, + { "snf", ContentType.Snf }, + { "so", ContentType.So }, + { "spc", ContentType.Spc }, + { "spdx", ContentType.Spdx }, + { "spf", ContentType.Spf }, + { "spl", ContentType.Spl }, + { "spot", ContentType.Spot }, + { "spp", ContentType.Spp }, + { "spq", ContentType.Spq }, + { "spx", ContentType.Spx }, + { "sql", ContentType.Sql }, + { "src", ContentType.Src }, + { "srt", ContentType.Srt }, + { "sru", ContentType.Sru }, + { "srx", ContentType.Srx }, + { "ssdl", ContentType.Ssdl }, + { "sse", ContentType.Sse }, + { "ssf", ContentType.Ssf }, + { "ssml", ContentType.Ssml }, + { "st", ContentType.St }, + { "stc", ContentType.Stc }, + { "std", ContentType.Std }, + { "stf", ContentType.Stf }, + { "sti", ContentType.Sti }, + { "stk", ContentType.Stk }, + { "stl", ContentType.Stl }, + { "stpx", ContentType.Stpx }, + { "stpxz", ContentType.Stpxz }, + { "stpz", ContentType.Stpz }, + { "str", ContentType.Str }, + { "stw", ContentType.Stw }, + { "styl", ContentType.Styl }, + { "stylus", ContentType.Stylus }, + { "sub", ContentType.Sub }, + { "sus", ContentType.Sus }, + { "susp", ContentType.Susp }, + { "sv4cpio", ContentType.Sv4cpio }, + { "sv4crc", ContentType.Sv4crc }, + { "svc", ContentType.Svc }, + { "svd", ContentType.Svd }, + { "svg", ContentType.Svg }, + { "svgz", ContentType.Svgz }, + { "swa", ContentType.Swa }, + { "swf", ContentType.Swf }, + { "swi", ContentType.Swi }, + { "swidtag", ContentType.Swidtag }, + { "sxc", ContentType.Sxc }, + { "sxd", ContentType.Sxd }, + { "sxg", ContentType.Sxg }, + { "sxi", ContentType.Sxi }, + { "sxm", ContentType.Sxm }, + { "sxw", ContentType.Sxw }, + { "t", ContentType.T }, + { "t3", ContentType.T3 }, + { "t38", ContentType.T38 }, + { "taglet", ContentType.Taglet }, + { "tao", ContentType.Tao }, + { "tap", ContentType.Tap }, + { "tar", ContentType.Tar }, + { "tcap", ContentType.Tcap }, + { "tcl", ContentType.Tcl }, + { "td", ContentType.Td }, + { "teacher", ContentType.Teacher }, + { "tei", ContentType.Tei }, + { "tex", ContentType.Tex }, + { "texi", ContentType.Texi }, + { "texinfo", ContentType.Texinfo }, + { "text", ContentType.Text }, + { "tfi", ContentType.Tfi }, + { "tfm", ContentType.Tfm }, + { "tfx", ContentType.Tfx }, + { "tga", ContentType.Tga }, + { "thmx", ContentType.Thmx }, + { "tif", ContentType.Tif }, + { "tiff", ContentType.Tiff }, + { "tk", ContentType.Tk }, + { "tmo", ContentType.Tmo }, + { "toml", ContentType.Toml }, + { "torrent", ContentType.Torrent }, + { "tpl", ContentType.Tpl }, + { "tpt", ContentType.Tpt }, + { "tr", ContentType.Tr }, + { "tra", ContentType.Tra }, + { "trig", ContentType.Trig }, + { "trm", ContentType.Trm }, + { "ts", ContentType.Ts }, + { "tsd", ContentType.Tsd }, + { "tsv", ContentType.Tsv }, + { "ttc", ContentType.Ttc }, + { "ttf", ContentType.Ttf }, + { "ttl", ContentType.Ttl }, + { "ttml", ContentType.Ttml }, + { "twd", ContentType.Twd }, + { "twds", ContentType.Twds }, + { "txd", ContentType.Txd }, + { "txf", ContentType.Txf }, + { "txt", ContentType.Txt }, + { "u32", ContentType.U32 }, + { "u8dsn", ContentType.U8dsn }, + { "u8hdr", ContentType.U8hdr }, + { "u8mdn", ContentType.U8mdn }, + { "u8msg", ContentType.U8msg }, + { "ubj", ContentType.Ubj }, + { "udeb", ContentType.Udeb }, + { "ufd", ContentType.Ufd }, + { "ufdl", ContentType.Ufdl }, + { "ulx", ContentType.Ulx }, + { "umj", ContentType.Umj }, + { "unityweb", ContentType.Unityweb }, + { "uoml", ContentType.Uoml }, + { "uri", ContentType.Uri }, + { "uris", ContentType.Uris }, + { "urls", ContentType.Urls }, + { "usdz", ContentType.Usdz }, + { "ustar", ContentType.Ustar }, + { "utz", ContentType.Utz }, + { "uu", ContentType.Uu }, + { "uva", ContentType.Uva }, + { "uvd", ContentType.Uvd }, + { "uvf", ContentType.Uvf }, + { "uvg", ContentType.Uvg }, + { "uvh", ContentType.Uvh }, + { "uvi", ContentType.Uvi }, + { "uvm", ContentType.Uvm }, + { "uvp", ContentType.Uvp }, + { "uvs", ContentType.Uvs }, + { "uvt", ContentType.Uvt }, + { "uvu", ContentType.Uvu }, + { "uvv", ContentType.Uvv }, + { "uvva", ContentType.Uvva }, + { "uvvd", ContentType.Uvvd }, + { "uvvf", ContentType.Uvvf }, + { "uvvg", ContentType.Uvvg }, + { "uvvh", ContentType.Uvvh }, + { "uvvi", ContentType.Uvvi }, + { "uvvm", ContentType.Uvvm }, + { "uvvp", ContentType.Uvvp }, + { "uvvs", ContentType.Uvvs }, + { "uvvt", ContentType.Uvvt }, + { "uvvu", ContentType.Uvvu }, + { "uvvv", ContentType.Uvvv }, + { "uvvx", ContentType.Uvvx }, + { "uvvz", ContentType.Uvvz }, + { "uvx", ContentType.Uvx }, + { "uvz", ContentType.Uvz }, + { "vbox", ContentType.Vbox }, + { "vcard", ContentType.Vcard }, + { "vcd", ContentType.Vcd }, + { "vcf", ContentType.Vcf }, + { "vcg", ContentType.Vcg }, + { "vcs", ContentType.Vcs }, + { "vcx", ContentType.Vcx }, + { "vdi", ContentType.Vdi }, + { "vds", ContentType.Vds }, + { "vhd", ContentType.Vhd }, + { "vis", ContentType.Vis }, + { "viv", ContentType.Viv }, + { "vmdk", ContentType.Vmdk }, + { "vob", ContentType.Vob }, + { "vor", ContentType.Vor }, + { "vox", ContentType.Vox }, + { "vrml", ContentType.Vrml }, + { "vsd", ContentType.Vsd }, + { "vsf", ContentType.Vsf }, + { "vss", ContentType.Vss }, + { "vst", ContentType.Vst }, + { "vsw", ContentType.Vsw }, + { "vtf", ContentType.Vtf }, + { "vtt", ContentType.Vtt }, + { "vtu", ContentType.Vtu }, + { "vxml", ContentType.Vxml }, + { "w3d", ContentType.W3d }, + { "wad", ContentType.Wad }, + { "wadl", ContentType.Wadl }, + { "war", ContentType.War }, + { "wasm", ContentType.Wasm }, + { "wav", ContentType.Wav }, + { "wax", ContentType.Wax }, + { "wbmp", ContentType.Wbmp }, + { "wbs", ContentType.Wbs }, + { "wbxml", ContentType.Wbxml }, + { "wcm", ContentType.Wcm }, + { "wdb", ContentType.Wdb }, + { "wdp", ContentType.Wdp }, + { "weba", ContentType.Weba }, + { "webapp", ContentType.Webapp }, + { "webm", ContentType.Webm }, + { "webp", ContentType.Webp }, + { "wg", ContentType.Wg }, + { "wgt", ContentType.Wgt }, + { "wks", ContentType.Wks }, + { "wm", ContentType.Wm }, + { "wma", ContentType.Wma }, + { "wmd", ContentType.Wmd }, + { "wmf", ContentType.Wmf }, + { "wml", ContentType.Wml }, + { "wmlc", ContentType.Wmlc }, + { "wmls", ContentType.Wmls }, + { "wmlsc", ContentType.Wmlsc }, + { "wmv", ContentType.Wmv }, + { "wmx", ContentType.Wmx }, + { "wmz", ContentType.Wmz }, + { "woff", ContentType.Woff }, + { "woff2", ContentType.Woff2 }, + { "wpd", ContentType.Wpd }, + { "wpl", ContentType.Wpl }, + { "wps", ContentType.Wps }, + { "wqd", ContentType.Wqd }, + { "wri", ContentType.Wri }, + { "wrl", ContentType.Wrl }, + { "wsc", ContentType.Wsc }, + { "wsdl", ContentType.Wsdl }, + { "wspolicy", ContentType.Wspolicy }, + { "wtb", ContentType.Wtb }, + { "wvx", ContentType.Wvx }, + { "x32", ContentType.X32 }, + { "x3d", ContentType.X3d }, + { "x3db", ContentType.X3db }, + { "x3dbz", ContentType.X3dbz }, + { "x3dv", ContentType.X3dv }, + { "x3dvz", ContentType.X3dvz }, + { "x3dz", ContentType.X3dz }, + { "xaml", ContentType.Xaml }, + { "xap", ContentType.Xap }, + { "xar", ContentType.Xar }, + { "xav", ContentType.Xav }, + { "xbap", ContentType.Xbap }, + { "xbd", ContentType.Xbd }, + { "xbm", ContentType.Xbm }, + { "xca", ContentType.Xca }, + { "xcs", ContentType.Xcs }, + { "xdf", ContentType.Xdf }, + { "xdm", ContentType.Xdm }, + { "xdp", ContentType.Xdp }, + { "xdssc", ContentType.Xdssc }, + { "xdw", ContentType.Xdw }, + { "xel", ContentType.Xel }, + { "xenc", ContentType.Xenc }, + { "xer", ContentType.Xer }, + { "xfdf", ContentType.Xfdf }, + { "xfdl", ContentType.Xfdl }, + { "xht", ContentType.Xht }, + { "xhtml", ContentType.Xhtml }, + { "xhvml", ContentType.Xhvml }, + { "xif", ContentType.Xif }, + { "xla", ContentType.Xla }, + { "xlam", ContentType.Xlam }, + { "xlc", ContentType.Xlc }, + { "xlf", ContentType.Xlf }, + { "xlm", ContentType.Xlm }, + { "xls", ContentType.Xls }, + { "xlsb", ContentType.Xlsb }, + { "xlsm", ContentType.Xlsm }, + { "xlsx", ContentType.Xlsx }, + { "xlt", ContentType.Xlt }, + { "xltm", ContentType.Xltm }, + { "xltx", ContentType.Xltx }, + { "xlw", ContentType.Xlw }, + { "xm", ContentType.Xm }, + { "xml", ContentType.Xml }, + { "xns", ContentType.Xns }, + { "xo", ContentType.Xo }, + { "xop", ContentType.Xop }, + { "xpi", ContentType.Xpi }, + { "xpl", ContentType.Xpl }, + { "xpm", ContentType.Xpm }, + { "xpr", ContentType.Xpr }, + { "xps", ContentType.Xps }, + { "xpw", ContentType.Xpw }, + { "xpx", ContentType.Xpx }, + { "xsd", ContentType.Xsd }, + { "xsl", ContentType.Xsl }, + { "xslt", ContentType.Xslt }, + { "xsm", ContentType.Xsm }, + { "xspf", ContentType.Xspf }, + { "xul", ContentType.Xul }, + { "xvm", ContentType.Xvm }, + { "xvml", ContentType.Xvml }, + { "xwd", ContentType.Xwd }, + { "xyz", ContentType.Xyz }, + { "xz", ContentType.Xz }, + { "yaml", ContentType.Yaml }, + { "yang", ContentType.Yang }, + { "yin", ContentType.Yin }, + { "yml", ContentType.Yml }, + { "ymp", ContentType.Ymp }, + { "z1", ContentType.Z1 }, + { "z2", ContentType.Z2 }, + { "z3", ContentType.Z3 }, + { "z4", ContentType.Z4 }, + { "z5", ContentType.Z5 }, + { "z6", ContentType.Z6 }, + { "z7", ContentType.Z7 }, + { "z8", ContentType.Z8 }, + { "zaz", ContentType.Zaz }, + { "zip", ContentType.Zip }, + { "zir", ContentType.Zir }, + { "zirz", ContentType.Zirz }, + { "zmm", ContentType.Zmm }, + }; + private static readonly IReadOnlyDictionary<string, ContentType> MimeToCt = new Dictionary<string, ContentType>() + { + { "application/x-www-form-urlencoded", ContentType.UrlEncoded }, + { "multipart/form-data", ContentType.MultiPart }, + { "audio/x-aac", ContentType.Aac }, + { "application/x-authorware-map", ContentType.Aam }, + { "application/x-authorware-seg", ContentType.Aas }, + { "application/x-abiword", ContentType.Abw }, + { "application/pkix-attr-cert", ContentType.Ac }, + { "application/vnd.americandynamics.acc", ContentType.Acc }, + { "application/x-ace-compressed", ContentType.Ace }, + { "application/vnd.acucobol", ContentType.Acu }, + { "application/vnd.acucorp", ContentType.Acutc }, + { "audio/adpcm", ContentType.Adp }, + { "application/vnd.audiograph", ContentType.Aep }, + { "application/vnd.ibm.modcap", ContentType.Afp }, + { "application/vnd.ahead.space", ContentType.Ahead }, + { "audio/x-aiff", ContentType.Aiff }, + { "application/vnd.adobe.air-application-installer-package+zip", ContentType.Air }, + { "application/vnd.dvb.ait", ContentType.Ait }, + { "application/vnd.amiga.ami", ContentType.Ami }, + { "audio/amr", ContentType.Amr }, + { "application/vnd.android.package-archive", ContentType.Apk }, + { "image/apng", ContentType.Apng }, + { "application/vnd.lotus-approach", ContentType.Apr }, + { "application/x-freearc", ContentType.Arc }, + { "application/x-arj", ContentType.Arj }, + { "video/x-ms-asf", ContentType.Asf }, + { "text/x-asm", ContentType.Asm }, + { "application/vnd.accpac.simply.aso", ContentType.Aso }, + { "application/atom+xml", ContentType.Atom }, + { "application/atomcat+xml", ContentType.Atomcat }, + { "application/atomsvc+xml", ContentType.Atomsvc }, + { "application/vnd.antix.game-component", ContentType.Atx }, + { "audio/basic", ContentType.Au }, + { "video/x-msvideo", ContentType.Avi }, + { "image/avif", ContentType.Avif }, + { "application/applixware", ContentType.Aw }, + { "application/vnd.airzip.filesecure.azf", ContentType.Azf }, + { "application/vnd.airzip.filesecure.azs", ContentType.Azs }, + { "image/vnd.airzip.accelerator.azv", ContentType.Azv }, + { "application/vnd.amazon.ebook", ContentType.Azw }, + { "image/vnd.pco.b16", ContentType.B16 }, + { "application/x-msdownload", ContentType.Bat }, + { "application/x-bcpio", ContentType.Bcpio }, + { "application/x-font-bdf", ContentType.Bdf }, + { "application/vnd.syncml.dm+wbxml", ContentType.Bdm }, + { "application/bdoc", ContentType.Bdoc }, + { "application/vnd.realvnc.bed", ContentType.Bed }, + { "application/vnd.fujitsu.oasysprs", ContentType.Bh2 }, + { "application/octet-stream", ContentType.Binary }, + { "application/x-blorb", ContentType.Blorb }, + { "application/vnd.bmi", ContentType.Bmi }, + { "application/vnd.balsamiq.bmml+xml", ContentType.Bmml }, + { "image/bmp", ContentType.Bmp }, + { "application/vnd.previewsystems.box", ContentType.Box }, + { "application/x-bzip2", ContentType.Boz }, + { "model/vnd.valve.source.compiled-map", ContentType.Bsp }, + { "image/prs.btif", ContentType.Btif }, + { "application/x-bzip", ContentType.Bz }, + { "text/x-c", ContentType.C }, + { "application/vnd.cluetrust.cartomobile-config", ContentType.C11amc }, + { "application/vnd.cluetrust.cartomobile-config-pkg", ContentType.C11amz }, + { "application/vnd.clonk.c4group", ContentType.C4d }, + { "application/vnd.ms-cab-compressed", ContentType.Cab }, + { "audio/x-caf", ContentType.Caf }, + { "application/vnd.curl.car", ContentType.Car }, + { "application/vnd.ms-pki.seccat", ContentType.Cat }, + { "application/x-cbr", ContentType.Cb7 }, + { "application/x-cocoa", ContentType.Cco }, + { "application/ccxml+xml", ContentType.Ccxml }, + { "application/vnd.contact.cmsg", ContentType.Cdbcmsg }, + { "application/x-netcdf", ContentType.Cdf }, + { "application/cdfx+xml", ContentType.Cdfx }, + { "application/vnd.mediastation.cdkey", ContentType.Cdkey }, + { "application/cdmi-capability", ContentType.Cdmia }, + { "application/cdmi-container", ContentType.Cdmic }, + { "application/cdmi-domain", ContentType.Cdmid }, + { "application/cdmi-object", ContentType.Cdmio }, + { "application/cdmi-queue", ContentType.Cdmiq }, + { "chemical/x-cdx", ContentType.Cdx }, + { "application/vnd.chemdraw+xml", ContentType.Cdxml }, + { "application/vnd.cinderella", ContentType.Cdy }, + { "application/pkix-cert", ContentType.Cer }, + { "application/x-cfs-compressed", ContentType.Cfs }, + { "image/cgm", ContentType.Cgm }, + { "application/x-chat", ContentType.Chat }, + { "application/vnd.ms-htmlhelp", ContentType.Chm }, + { "application/vnd.kde.kchart", ContentType.Chrt }, + { "chemical/x-cif", ContentType.Cif }, + { "application/vnd.anser-web-certificate-issue-initiation", ContentType.Cii }, + { "application/vnd.ms-artgalry", ContentType.Cil }, + { "application/node", ContentType.Cjs }, + { "application/vnd.claymore", ContentType.Cla }, + { "application/vnd.crick.clicker.keyboard", ContentType.Clkk }, + { "application/vnd.crick.clicker.palette", ContentType.Clkp }, + { "application/vnd.crick.clicker.template", ContentType.Clkt }, + { "application/vnd.crick.clicker.wordbank", ContentType.Clkw }, + { "application/vnd.crick.clicker", ContentType.Clkx }, + { "application/x-msclip", ContentType.Clp }, + { "application/vnd.cosmocaller", ContentType.Cmc }, + { "chemical/x-cmdf", ContentType.Cmdf }, + { "chemical/x-cml", ContentType.Cml }, + { "application/vnd.yellowriver-custom-menu", ContentType.Cmp }, + { "image/x-cmx", ContentType.Cmx }, + { "application/vnd.rim.cod", ContentType.Cod }, + { "text/coffeescript", ContentType.Coffee }, + { "application/x-cpio", ContentType.Cpio }, + { "application/mac-compactpro", ContentType.Cpt }, + { "application/x-mscardfile", ContentType.Crd }, + { "application/pkix-crl", ContentType.Crl }, + { "application/x-x509-ca-cert", ContentType.Crt }, + { "application/x-chrome-extension", ContentType.Crx }, + { "application/x-csh", ContentType.Csh }, + { "application/vnd.citationstyles.style+xml", ContentType.Csl }, + { "chemical/x-csml", ContentType.Csml }, + { "application/vnd.commonspace", ContentType.Csp }, + { "text/css", ContentType.Css }, + { "text/csv", ContentType.Csv }, + { "application/cu-seeme", ContentType.Cu }, + { "text/vnd.curl", ContentType.Curl }, + { "application/prs.cww", ContentType.Cww }, + { "model/vnd.collada+xml", ContentType.Dae }, + { "application/vnd.mobius.daf", ContentType.Daf }, + { "application/vnd.dart", ContentType.Dart }, + { "application/davmount+xml", ContentType.Davmount }, + { "application/vnd.dbf", ContentType.Dbf }, + { "application/docbook+xml", ContentType.Dbk }, + { "application/x-director", ContentType.Dcr }, + { "text/vnd.curl.dcurl", ContentType.Dcurl }, + { "application/vnd.oma.dd2+xml", ContentType.Dd2 }, + { "application/vnd.fujixerox.ddd", ContentType.Ddd }, + { "application/vnd.syncml.dmddf+xml", ContentType.Ddf }, + { "image/vnd.ms-dds", ContentType.Dds }, + { "application/vnd.dreamfactory", ContentType.Dfac }, + { "application/x-dgc-compressed", ContentType.Dgc }, + { "application/vnd.mobius.dis", ContentType.Dis }, + { "image/vnd.djvu", ContentType.Djvu }, + { "application/vnd.dna", ContentType.Dna }, + { "application/msword", ContentType.Doc }, + { "application/vnd.ms-word.document.macroenabled.12", ContentType.Docm }, + { "application/vnd.openxmlformats-officedocument.wordprocessingml.document", ContentType.Docx }, + { "application/vnd.openxmlformats-officedocument.wordprocessingml.template", ContentType.Dotx }, + { "application/vnd.osgi.dp", ContentType.Dp }, + { "application/vnd.dpgraph", ContentType.Dpg }, + { "audio/vnd.dra", ContentType.Dra }, + { "image/dicom-rle", ContentType.Drle }, + { "text/prs.lines.tag", ContentType.Dsc }, + { "application/dssc+der", ContentType.Dssc }, + { "application/x-dtbook+xml", ContentType.Dtb }, + { "application/xml-dtd", ContentType.Dtd }, + { "audio/vnd.dts", ContentType.Dts }, + { "audio/vnd.dts.hd", ContentType.Dtshd }, + { "video/vnd.dvb.file", ContentType.Dvb }, + { "application/x-dvi", ContentType.Dvi }, + { "application/atsc-dwd+xml", ContentType.Dwd }, + { "model/vnd.dwf", ContentType.Dwf }, + { "image/vnd.dwg", ContentType.Dwg }, + { "image/vnd.dxf", ContentType.Dxf }, + { "application/vnd.spotfire.dxp", ContentType.Dxp }, + { "application/ecmascript", ContentType.Ecma }, + { "application/vnd.novadigm.edm", ContentType.Edm }, + { "application/vnd.novadigm.edx", ContentType.Edx }, + { "application/vnd.picsel", ContentType.Efif }, + { "application/vnd.pg.osasli", ContentType.Ei6 }, + { "application/emma+xml", ContentType.Emma }, + { "audio/vnd.digital-winds", ContentType.Eol }, + { "application/vnd.ms-fontobject", ContentType.Eot }, + { "application/epub+zip", ContentType.Epub }, + { "application/vnd.osgi.subsystem", ContentType.Esa }, + { "application/vnd.epson.esf", ContentType.Esf }, + { "application/vnd.eszigno3+xml", ContentType.Et3 }, + { "text/x-setext", ContentType.Etx }, + { "application/x-eva", ContentType.Eva }, + { "application/x-envoy", ContentType.Evy }, + { "application/exi", ContentType.Exi }, + { "application/express", ContentType.Exp }, + { "image/aces", ContentType.Exr }, + { "application/vnd.novadigm.ext", ContentType.Ext }, + { "application/andrew-inset", ContentType.Ez }, + { "application/vnd.ezpix-album", ContentType.Ez2 }, + { "application/vnd.ezpix-package", ContentType.Ez3 }, + { "video/x-f4v", ContentType.F4v }, + { "text/x-fortran", ContentType.Fortran }, + { "image/vnd.fastbidsheet", ContentType.Fbs }, + { "application/vnd.adobe.formscentral.fcdt", ContentType.Fcdt }, + { "application/vnd.isac.fcs", ContentType.Fcs }, + { "application/vnd.fdf", ContentType.Fdf }, + { "application/fdt+xml", ContentType.Fdt }, + { "application/vnd.fujitsu.oasysgp", ContentType.Fg5 }, + { "image/x-freehand", ContentType.Fh }, + { "application/x-xfig", ContentType.Fig }, + { "image/fits", ContentType.Fits }, + { "audio/x-flac", ContentType.Flac }, + { "video/x-fli", ContentType.Fli }, + { "application/vnd.micrografx.flo", ContentType.Flo }, + { "video/x-flv", ContentType.Flv }, + { "application/vnd.kde.kivio", ContentType.Flw }, + { "text/vnd.fmi.flexstor", ContentType.Flx }, + { "text/vnd.fly", ContentType.Fly }, + { "application/vnd.frogans.fnc", ContentType.Fnc }, + { "application/vnd.software602.filler.form+xml", ContentType.Fo }, + { "image/vnd.fpx", ContentType.Fpx }, + { "application/vnd.framemaker", ContentType.Frame }, + { "application/vnd.fsc.weblaunch", ContentType.Fsc }, + { "image/vnd.fst", ContentType.Fst }, + { "application/vnd.fluxtime.clip", ContentType.Ftc }, + { "application/vnd.anser-web-funds-transfer-initiation", ContentType.Fti }, + { "video/vnd.fvt", ContentType.Fvt }, + { "application/vnd.adobe.fxp", ContentType.Fxp }, + { "application/vnd.fuzzysheet", ContentType.Fzs }, + { "application/vnd.geoplan", ContentType.G2w }, + { "image/g3fax", ContentType.G3 }, + { "application/vnd.geospace", ContentType.G3w }, + { "application/vnd.groove-account", ContentType.Gac }, + { "application/x-tads", ContentType.Gam }, + { "application/rpki-ghostbusters", ContentType.Gbr }, + { "application/x-gca-compressed", ContentType.Gca }, + { "model/vnd.gdl", ContentType.Gdl }, + { "application/vnd.google-apps.document", ContentType.Gdoc }, + { "application/vnd.dynageo", ContentType.Geo }, + { "application/geo+json", ContentType.Geojson }, + { "application/vnd.geometry-explorer", ContentType.Gex }, + { "application/vnd.geogebra.file", ContentType.Ggb }, + { "application/vnd.geogebra.tool", ContentType.Ggt }, + { "application/vnd.groove-help", ContentType.Ghf }, + { "image/gif", ContentType.Gif }, + { "application/vnd.groove-identity-message", ContentType.Gim }, + { "model/gltf-binary", ContentType.Glb }, + { "model/gltf+json", ContentType.Gltf }, + { "application/gml+xml", ContentType.Gml }, + { "application/vnd.gmx", ContentType.Gmx }, + { "application/x-gnumeric", ContentType.Gnumeric }, + { "application/vnd.flographit", ContentType.Gph }, + { "application/gpx+xml", ContentType.Gpx }, + { "application/vnd.grafeq", ContentType.Gqf }, + { "application/srgs", ContentType.Gram }, + { "application/x-gramps-xml", ContentType.Gramps }, + { "application/vnd.groove-injector", ContentType.Grv }, + { "application/srgs+xml", ContentType.Grxml }, + { "application/x-font-ghostscript", ContentType.Gsf }, + { "application/vnd.google-apps.spreadsheet", ContentType.Gsheet }, + { "application/vnd.google-apps.presentation", ContentType.Gslides }, + { "application/x-gtar", ContentType.Gtar }, + { "application/vnd.groove-tool-message", ContentType.Gtm }, + { "model/vnd.gtw", ContentType.Gtw }, + { "text/vnd.graphviz", ContentType.Gv }, + { "application/gxf", ContentType.Gxf }, + { "application/vnd.geonext", ContentType.Gxt }, + { "application/gzip", ContentType.Gz }, + { "video/h261", ContentType.H261 }, + { "video/h263", ContentType.H263 }, + { "video/h264", ContentType.H264 }, + { "application/vnd.hal+xml", ContentType.Hal }, + { "application/vnd.hbci", ContentType.Hbci }, + { "text/x-handlebars-template", ContentType.Hbs }, + { "application/x-virtualbox-hdd", ContentType.Hdd }, + { "application/x-hdf", ContentType.Hdf }, + { "image/heic", ContentType.Heic }, + { "image/heic-sequence", ContentType.Heics }, + { "image/heif", ContentType.Heif }, + { "image/heif-sequence", ContentType.Heifs }, + { "image/hej2k", ContentType.Hej2 }, + { "application/atsc-held+xml", ContentType.Held }, + { "application/hjson", ContentType.Hjson }, + { "application/winhlp", ContentType.Hlp }, + { "application/vnd.hp-hpgl", ContentType.Hpgl }, + { "application/vnd.hp-hpid", ContentType.Hpid }, + { "application/vnd.hp-hps", ContentType.Hps }, + { "application/mac-binhex40", ContentType.Hqx }, + { "image/hsj2", ContentType.Hsj2 }, + { "application/vnd.kenameaapp", ContentType.Htke }, + { "text/html", ContentType.Html }, + { "application/vnd.yamaha.hv-dic", ContentType.Hvd }, + { "application/vnd.yamaha.hv-voice", ContentType.Hvp }, + { "application/vnd.yamaha.hv-script", ContentType.Hvs }, + { "application/vnd.intergeo", ContentType.I2g }, + { "application/vnd.iccprofile", ContentType.Icc }, + { "x-conference/x-cooltalk", ContentType.Ice }, + { "image/vnd.microsoft.icon", ContentType.Ico }, + { "image/ief", ContentType.Ief }, + { "application/vnd.shana.informed.formdata", ContentType.Ifm }, + { "model/iges", ContentType.Iges }, + { "application/vnd.igloader", ContentType.Igl }, + { "application/vnd.insors.igm", ContentType.Igm }, + { "application/vnd.micrografx.igx", ContentType.Igx }, + { "application/vnd.shana.informed.interchange", ContentType.Iif }, + { "application/vnd.accpac.simply.imp", ContentType.Imp }, + { "application/vnd.ms-ims", ContentType.Ims }, + { "application/inkml+xml", ContentType.Inkml }, + { "application/x-install-instructions", ContentType.Install }, + { "application/vnd.astraea-software.iota", ContentType.Iota }, + { "application/ipfix", ContentType.Ipfix }, + { "application/vnd.shana.informed.package", ContentType.Ipk }, + { "application/vnd.ibm.rights-management", ContentType.Irm }, + { "application/vnd.irepository.package+xml", ContentType.Irp }, + { "application/vnd.shana.informed.formtemplate", ContentType.Itp }, + { "application/its+xml", ContentType.Its }, + { "application/vnd.immervision-ivp", ContentType.Ivp }, + { "application/vnd.immervision-ivu", ContentType.Ivu }, + { "text/vnd.sun.j2me.app-descriptor", ContentType.Jad }, + { "text/jade", ContentType.Jade }, + { "application/vnd.jam", ContentType.Jam }, + { "application/java-archive", ContentType.Jar }, + { "application/x-java-archive-diff", ContentType.Jardiff }, + { "text/x-java-source", ContentType.Java }, + { "image/jphc", ContentType.Jhc }, + { "application/vnd.jisp", ContentType.Jisp }, + { "image/jls", ContentType.Jls }, + { "application/vnd.hp-jlyt", ContentType.Jlt }, + { "image/x-jng", ContentType.Jng }, + { "application/x-java-jnlp-file", ContentType.Jnlp }, + { "application/vnd.joost.joda-archive", ContentType.Joda }, + { "image/jp2", ContentType.Jp2 }, + { "image/jpeg", ContentType.Jpeg }, + { "video/jpm", ContentType.Jpgm }, + { "video/jpeg", ContentType.Jpgv }, + { "image/jph", ContentType.Jph }, + { "image/jpm", ContentType.Jpm }, + { "image/jpx", ContentType.Jpx }, + { "application/javascript", ContentType.Javascript }, + { "application/json", ContentType.Json }, + { "application/json5", ContentType.Json5 }, + { "application/ld+json", ContentType.Jsonld }, + { "application/jsonml+json", ContentType.Jsonml }, + { "text/jsx", ContentType.Jsx }, + { "image/jxr", ContentType.Jxr }, + { "image/jxra", ContentType.Jxra }, + { "image/jxrs", ContentType.Jxrs }, + { "image/jxs", ContentType.Jxs }, + { "image/jxsc", ContentType.Jxsc }, + { "image/jxsi", ContentType.Jxsi }, + { "image/jxss", ContentType.Jxss }, + { "application/vnd.kde.karbon", ContentType.Karbon }, + { "application/x-keepass2", ContentType.Kdbx }, + { "application/vnd.apple.keynote", ContentType.Key }, + { "application/vnd.kde.kformula", ContentType.Kfo }, + { "application/vnd.kidspiration", ContentType.Kia }, + { "application/vnd.google-earth.kml+xml", ContentType.Kml }, + { "application/vnd.google-earth.kmz", ContentType.Kmz }, + { "application/vnd.kinar", ContentType.Kne }, + { "application/vnd.kde.kontour", ContentType.Kon }, + { "application/vnd.kde.kpresenter", ContentType.Kpr }, + { "application/vnd.ds-keypoint", ContentType.Kpxx }, + { "application/vnd.kde.kspread", ContentType.Ksp }, + { "image/ktx", ContentType.Ktx }, + { "image/ktx2", ContentType.Ktx2 }, + { "application/vnd.kahootz", ContentType.Ktz }, + { "application/vnd.kde.kword", ContentType.Kwd }, + { "application/vnd.las.las+xml", ContentType.Lasxml }, + { "application/x-latex", ContentType.Latex }, + { "application/vnd.llamagraphics.life-balance.desktop", ContentType.Lbd }, + { "application/vnd.llamagraphics.life-balance.exchange+xml", ContentType.Lbe }, + { "application/vnd.hhe.lesson-player", ContentType.Les }, + { "text/less", ContentType.Less }, + { "application/lgr+xml", ContentType.Lgr }, + { "application/vnd.route66.link66+xml", ContentType.Link66 }, + { "application/x-ms-shortcut", ContentType.Lnk }, + { "application/lost+xml", ContentType.Lostxml }, + { "application/vnd.ms-lrm", ContentType.Lrm }, + { "application/vnd.frogans.ltf", ContentType.Ltf }, + { "text/x-lua", ContentType.Lua }, + { "application/x-lua-bytecode", ContentType.Luac }, + { "audio/vnd.lucent.voice", ContentType.Lvp }, + { "application/vnd.lotus-wordpro", ContentType.Lwp }, + { "application/x-lzh-compressed", ContentType.Lzh }, + { "audio/mpeg", ContentType.M2a }, + { "audio/x-mpegurl", ContentType.M3u }, + { "application/vnd.apple.mpegurl", ContentType.M3u8 }, + { "audio/mp4", ContentType.M4a }, + { "video/iso.segment", ContentType.M4s }, + { "video/vnd.mpegurl", ContentType.M4u }, + { "video/x-m4v", ContentType.M4v }, + { "application/mathematica", ContentType.Ma }, + { "application/mads+xml", ContentType.Mads }, + { "application/mmt-aei+xml", ContentType.Maei }, + { "application/vnd.ecowin.chart", ContentType.Mag }, + { "text/cache-manifest", ContentType.Manifest }, + { "text/markdown", ContentType.Markdown }, + { "application/mathml+xml", ContentType.Mathml }, + { "application/vnd.mobius.mbk", ContentType.Mbk }, + { "application/mbox", ContentType.Mbox }, + { "application/vnd.medcalcdata", ContentType.Mc1 }, + { "application/vnd.mcd", ContentType.Mcd }, + { "text/vnd.curl.mcurl", ContentType.Mcurl }, + { "application/x-msaccess", ContentType.Mdb }, + { "image/vnd.ms-modi", ContentType.Mdi }, + { "text/mdx", ContentType.Mdx }, + { "model/mesh", ContentType.Mesh }, + { "application/metalink4+xml", ContentType.Meta4 }, + { "application/metalink+xml", ContentType.Metalink }, + { "application/mets+xml", ContentType.Mets }, + { "application/vnd.mfmp", ContentType.Mfm }, + { "application/rpki-manifest", ContentType.Mft }, + { "application/vnd.osgeo.mapguide.package", ContentType.Mgp }, + { "application/vnd.proteus.magazine", ContentType.Mgz }, + { "audio/midi", ContentType.Midi }, + { "application/x-mie", ContentType.Mie }, + { "application/vnd.mif", ContentType.Mif }, + { "message/rfc822", ContentType.Mime }, + { "video/mj2", ContentType.Mj2 }, + { "audio/x-matroska", ContentType.Mka }, + { "text/x-markdown", ContentType.Mkd }, + { "video/x-matroska", ContentType.Mkv }, + { "application/vnd.dolby.mlp", ContentType.Mlp }, + { "application/vnd.chipnuts.karaoke-mmd", ContentType.Mmd }, + { "application/vnd.smaf", ContentType.Mmf }, + { "text/mathml", ContentType.Mml }, + { "image/vnd.fujixerox.edmics-mmr", ContentType.Mmr }, + { "video/x-mng", ContentType.Mng }, + { "application/x-msmoney", ContentType.Mny }, + { "application/mods+xml", ContentType.Mods }, + { "video/x-sgi-movie", ContentType.Movie }, + { "application/mp21", ContentType.Mp21 }, + { "audio/mp3", ContentType.Mp3 }, + { "video/mp4", ContentType.Mp4 }, + { "application/mp4", ContentType.Mp4s }, + { "application/vnd.mophun.certificate", ContentType.Mpc }, + { "application/dash+xml", ContentType.Mpd }, + { "video/mpeg", ContentType.Mpeg }, + { "application/vnd.apple.installer+xml", ContentType.Mpkg }, + { "application/vnd.blueice.multipass", ContentType.Mpm }, + { "application/vnd.mophun.application", ContentType.Mpn }, + { "application/vnd.ms-project", ContentType.Mpt }, + { "application/vnd.ibm.minipay", ContentType.Mpy }, + { "application/vnd.mobius.mqy", ContentType.Mqy }, + { "application/marc", ContentType.Mrc }, + { "application/marcxml+xml", ContentType.Mrcx }, + { "application/mediaservercontrol+xml", ContentType.Mscml }, + { "application/vnd.fdsn.mseed", ContentType.Mseed }, + { "application/vnd.mseq", ContentType.Mseq }, + { "application/vnd.epson.msf", ContentType.Msf }, + { "application/vnd.ms-outlook", ContentType.Msg }, + { "application/vnd.mobius.msl", ContentType.Msl }, + { "application/vnd.muvee.style", ContentType.Msty }, + { "model/mtl", ContentType.Mtl }, + { "model/vnd.mts", ContentType.Mts }, + { "application/vnd.musician", ContentType.Mus }, + { "application/mmt-usd+xml", ContentType.Musd }, + { "application/vnd.recordare.musicxml+xml", ContentType.Musicxml }, + { "application/x-msmediaview", ContentType.Mvb }, + { "application/vnd.mapbox-vector-tile", ContentType.Mvt }, + { "application/vnd.mfer", ContentType.Mwf }, + { "application/mxf", ContentType.Mxf }, + { "application/vnd.recordare.musicxml", ContentType.Mxl }, + { "audio/mobile-xmf", ContentType.Mxmf }, + { "application/vnd.triscape.mxs", ContentType.Mxs }, + { "text/n3", ContentType.N3 }, + { "application/vnd.wolfram.player", ContentType.Nbp }, + { "application/x-dtbncx+xml", ContentType.Ncx }, + { "text/x-nfo", ContentType.Nfo }, + { "application/vnd.nokia.n-gage.data", ContentType.Ngdat }, + { "application/vnd.nitf", ContentType.Nitf }, + { "application/vnd.neurolanguage.nlu", ContentType.Nlu }, + { "application/vnd.enliven", ContentType.Nml }, + { "application/vnd.noblenet-directory", ContentType.Nnd }, + { "application/vnd.noblenet-sealer", ContentType.Nns }, + { "application/vnd.noblenet-web", ContentType.Nnw }, + { "image/vnd.net-fpx", ContentType.Npx }, + { "application/n-quads", ContentType.Nq }, + { "application/x-conference", ContentType.Nsc }, + { "application/vnd.lotus-notes", ContentType.Nsf }, + { "application/n-triples", ContentType.Nt }, + { "application/vnd.apple.numbers", ContentType.Numbers }, + { "application/x-nzb", ContentType.Nzb }, + { "application/vnd.fujitsu.oasys2", ContentType.Oa2 }, + { "application/vnd.fujitsu.oasys3", ContentType.Oa3 }, + { "application/vnd.fujitsu.oasys", ContentType.Oas }, + { "application/x-msbinder", ContentType.Obd }, + { "application/vnd.openblox.game+xml", ContentType.Obgx }, + { "application/x-tgif", ContentType.Obj }, + { "application/oda", ContentType.Oda }, + { "application/vnd.oasis.opendocument.database", ContentType.Odb }, + { "application/vnd.oasis.opendocument.chart", ContentType.Odc }, + { "application/vnd.oasis.opendocument.formula", ContentType.Odf }, + { "application/vnd.oasis.opendocument.formula-template", ContentType.Odft }, + { "application/vnd.oasis.opendocument.graphics", ContentType.Odg }, + { "application/vnd.oasis.opendocument.image", ContentType.Odi }, + { "application/vnd.oasis.opendocument.text-master", ContentType.Odm }, + { "application/vnd.oasis.opendocument.presentation", ContentType.Odp }, + { "application/vnd.oasis.opendocument.spreadsheet", ContentType.Ods }, + { "application/vnd.oasis.opendocument.text", ContentType.Odt }, + { "model/vnd.opengex", ContentType.Ogex }, + { "audio/ogg", ContentType.Ogg }, + { "video/ogg", ContentType.Ogv }, + { "application/ogg", ContentType.Ogx }, + { "application/omdoc+xml", ContentType.Omdoc }, + { "application/onenote", ContentType.Onetoc }, + { "application/oebps-package+xml", ContentType.Opf }, + { "text/x-opml", ContentType.Opml }, + { "application/vnd.lotus-organizer", ContentType.Org }, + { "application/vnd.yamaha.openscoreformat", ContentType.Osf }, + { "application/vnd.yamaha.openscoreformat.osfpvg+xml", ContentType.Osfpvg }, + { "application/vnd.openstreetmap.data+xml", ContentType.Osm }, + { "application/vnd.oasis.opendocument.chart-template", ContentType.Otc }, + { "font/otf", ContentType.Otf }, + { "application/vnd.oasis.opendocument.graphics-template", ContentType.Otg }, + { "application/vnd.oasis.opendocument.text-web", ContentType.Oth }, + { "application/vnd.oasis.opendocument.image-template", ContentType.Oti }, + { "application/vnd.oasis.opendocument.presentation-template", ContentType.Otp }, + { "application/vnd.oasis.opendocument.spreadsheet-template", ContentType.Ots }, + { "application/vnd.oasis.opendocument.text-template", ContentType.Ott }, + { "application/x-virtualbox-ova", ContentType.Ova }, + { "application/x-virtualbox-ovf", ContentType.Ovf }, + { "application/oxps", ContentType.Oxps }, + { "application/vnd.openofficeorg.extension", ContentType.Oxt }, + { "text/x-pascal", ContentType.P }, + { "application/pkcs10", ContentType.P10 }, + { "application/x-pkcs7-certificates", ContentType.P7b }, + { "application/pkcs7-mime", ContentType.P7m }, + { "application/x-pkcs7-certreqresp", ContentType.P7r }, + { "application/pkcs7-signature", ContentType.P7s }, + { "application/pkcs8", ContentType.P8 }, + { "application/x-ns-proxy-autoconfig", ContentType.Pac }, + { "application/vnd.apple.pages", ContentType.Pages }, + { "application/vnd.pawaafile", ContentType.Paw }, + { "application/vnd.powerbuilder6", ContentType.Pbd }, + { "image/x-portable-bitmap", ContentType.Pbm }, + { "application/vnd.tcpdump.pcap", ContentType.Pcap }, + { "application/x-font-pcf", ContentType.Pcf }, + { "application/vnd.hp-pcl", ContentType.Pcl }, + { "application/vnd.hp-pclxl", ContentType.Pclxl }, + { "image/x-pict", ContentType.Pct }, + { "application/vnd.curl.pcurl", ContentType.Pcurl }, + { "image/vnd.zbrush.pcx", ContentType.Pcx }, + { "application/vnd.palm", ContentType.Pdb }, + { "text/x-processing", ContentType.Pde }, + { "application/pdf", ContentType.Pdf }, + { "application/x-font-type1", ContentType.Pfa }, + { "application/font-tdpfr", ContentType.Pfr }, + { "application/x-pkcs12", ContentType.Pfx }, + { "image/x-portable-graymap", ContentType.Pgm }, + { "application/x-chess-pgn", ContentType.Pgn }, + { "application/pgp-encrypted", ContentType.Pgp }, + { "application/x-httpd-php", ContentType.Php }, + { "application/pkixcmp", ContentType.Pki }, + { "application/pkix-pkipath", ContentType.Pkipath }, + { "application/vnd.apple.pkpass", ContentType.Pkpass }, + { "application/x-perl", ContentType.Pl }, + { "application/vnd.3gpp.pic-bw-large", ContentType.Plb }, + { "application/vnd.mobius.plc", ContentType.Plc }, + { "application/vnd.pocketlearn", ContentType.Plf }, + { "application/pls+xml", ContentType.Pls }, + { "application/vnd.ctc-posml", ContentType.Pml }, + { "image/png", ContentType.Png }, + { "image/x-portable-anymap", ContentType.Pnm }, + { "application/vnd.macports.portpkg", ContentType.Portpkg }, + { "application/vnd.ms-powerpoint.template.macroenabled.12", ContentType.Potm }, + { "application/vnd.openxmlformats-officedocument.presentationml.template", ContentType.Potx }, + { "application/vnd.ms-powerpoint.addin.macroenabled.12", ContentType.Ppam }, + { "application/vnd.cups-ppd", ContentType.Ppd }, + { "image/x-portable-pixmap", ContentType.Ppm }, + { "application/vnd.ms-powerpoint.slideshow.macroenabled.12", ContentType.Ppsm }, + { "application/vnd.openxmlformats-officedocument.presentationml.slideshow", ContentType.Ppsx }, + { "application/vnd.ms-powerpoint", ContentType.Ppt }, + { "application/vnd.ms-powerpoint.presentation.macroenabled.12", ContentType.Pptm }, + { "application/vnd.openxmlformats-officedocument.presentationml.presentation", ContentType.Pptx }, + { "application/x-mobipocket-ebook", ContentType.Prc }, + { "application/vnd.lotus-freelance", ContentType.Pre }, + { "application/pics-rules", ContentType.Prf }, + { "application/provenance+xml", ContentType.Provx }, + { "application/postscript", ContentType.Ps }, + { "application/vnd.3gpp.pic-bw-small", ContentType.Psb }, + { "image/vnd.adobe.photoshop", ContentType.Psd }, + { "application/x-font-linux-psf", ContentType.Psf }, + { "application/pskc+xml", ContentType.Pskcxml }, + { "image/prs.pti", ContentType.Pti }, + { "application/vnd.pvi.ptid1", ContentType.Ptid }, + { "application/x-mspublisher", ContentType.Pub }, + { "application/vnd.3gpp.pic-bw-var", ContentType.Pvb }, + { "application/vnd.3m.post-it-notes", ContentType.Pwn }, + { "audio/vnd.ms-playready.media.pya", ContentType.Pya }, + { "video/vnd.ms-playready.media.pyv", ContentType.Pyv }, + { "application/vnd.epson.quickanime", ContentType.Qam }, + { "application/vnd.intu.qbo", ContentType.Qbo }, + { "application/vnd.intu.qfx", ContentType.Qfx }, + { "application/vnd.publishare-delta-tree", ContentType.Qps }, + { "video/quicktime", ContentType.Qt }, + { "application/vnd.quark.quarkxpress", ContentType.Qwd }, + { "audio/x-pn-realaudio", ContentType.Ra }, + { "application/raml+yaml", ContentType.Raml }, + { "application/route-apd+xml", ContentType.Rapd }, + { "application/vnd.rar", ContentType.Rar }, + { "image/x-cmu-raster", ContentType.Ras }, + { "application/rdf+xml", ContentType.Rdf }, + { "application/vnd.data-vision.rdz", ContentType.Rdz }, + { "application/p2p-overlay+xml", ContentType.Relo }, + { "application/vnd.businessobjects", ContentType.Rep }, + { "application/x-dtbresource+xml", ContentType.Res }, + { "image/x-rgb", ContentType.Rgb }, + { "application/reginfo+xml", ContentType.Rif }, + { "audio/vnd.rip", ContentType.Rip }, + { "application/x-research-info-systems", ContentType.Ris }, + { "application/resource-lists+xml", ContentType.Rl }, + { "image/vnd.fujixerox.edmics-rlc", ContentType.Rlc }, + { "application/resource-lists-diff+xml", ContentType.Rld }, + { "application/vnd.rn-realmedia", ContentType.Rm }, + { "audio/x-pn-realaudio-plugin", ContentType.Rmp }, + { "application/vnd.jcp.javame.midlet-rms", ContentType.Rms }, + { "application/vnd.rn-realmedia-vbr", ContentType.Rmvb }, + { "application/relax-ng-compact-syntax", ContentType.Rnc }, + { "application/rpki-roa", ContentType.Roa }, + { "text/troff", ContentType.Roff }, + { "application/vnd.cloanto.rp9", ContentType.Rp9 }, + { "application/x-redhat-package-manager", ContentType.Rpm }, + { "application/vnd.nokia.radio-presets", ContentType.Rpss }, + { "application/vnd.nokia.radio-preset", ContentType.Rpst }, + { "application/sparql-query", ContentType.Rq }, + { "application/rls-services+xml", ContentType.Rs }, + { "application/atsc-rsat+xml", ContentType.Rsat }, + { "application/rsd+xml", ContentType.Rsd }, + { "application/urc-ressheet+xml", ContentType.Rsheet }, + { "application/rss+xml", ContentType.Rss }, + { "application/rtf", ContentType.Rtf }, + { "text/richtext", ContentType.Rtx }, + { "application/x-makeself", ContentType.Run }, + { "application/route-usd+xml", ContentType.Rusd }, + { "audio/s3m", ContentType.S3m }, + { "application/vnd.yamaha.smaf-audio", ContentType.Saf }, + { "text/x-sass", ContentType.Sass }, + { "application/sbml+xml", ContentType.Sbml }, + { "application/vnd.ibm.secure-container", ContentType.Sc }, + { "application/x-msschedule", ContentType.Scd }, + { "application/vnd.lotus-screencam", ContentType.Scm }, + { "application/scvp-cv-request", ContentType.Scq }, + { "application/scvp-cv-response", ContentType.Scs }, + { "text/x-scss", ContentType.Scss }, + { "text/vnd.curl.scurl", ContentType.Scurl }, + { "application/vnd.stardivision.draw", ContentType.Sda }, + { "application/vnd.stardivision.calc", ContentType.Sdc }, + { "application/vnd.stardivision.impress", ContentType.Sdd }, + { "application/vnd.solent.sdkm+xml", ContentType.Sdkm }, + { "application/sdp", ContentType.Sdp }, + { "application/vnd.stardivision.writer", ContentType.Sdw }, + { "application/x-sea", ContentType.Sea }, + { "application/vnd.seemail", ContentType.See }, + { "application/vnd.fdsn.seed", ContentType.Seed }, + { "application/vnd.sema", ContentType.Sema }, + { "application/vnd.semd", ContentType.Semd }, + { "application/vnd.semf", ContentType.Semf }, + { "application/senml+xml", ContentType.Senmlx }, + { "application/sensml+xml", ContentType.Sensmlx }, + { "application/java-serialized-object", ContentType.Ser }, + { "application/set-payment-initiation", ContentType.Setpay }, + { "application/set-registration-initiation", ContentType.Setreg }, + { "application/vnd.spotfire.sfs", ContentType.Sfs }, + { "text/x-sfv", ContentType.Sfv }, + { "image/sgi", ContentType.Sgi }, + { "application/vnd.stardivision.writer-global", ContentType.Sgl }, + { "text/sgml", ContentType.Sgml }, + { "application/x-sh", ContentType.Sh }, + { "application/x-shar", ContentType.Shar }, + { "text/shex", ContentType.Shex }, + { "application/shf+xml", ContentType.Shf }, + { "image/x-mrsid-image", ContentType.Sid }, + { "application/sieve", ContentType.Sieve }, + { "application/pgp-signature", ContentType.Sig }, + { "audio/silk", ContentType.Sil }, + { "application/vnd.symbian.install", ContentType.Sisx }, + { "application/x-stuffit", ContentType.Sit }, + { "application/x-stuffitx", ContentType.Sitx }, + { "application/vnd.koan", ContentType.Skd }, + { "application/vnd.ms-powerpoint.slide.macroenabled.12", ContentType.Sldm }, + { "application/vnd.openxmlformats-officedocument.presentationml.slide", ContentType.Sldx }, + { "text/slim", ContentType.Slim }, + { "application/route-s-tsid+xml", ContentType.Sls }, + { "application/vnd.epson.salt", ContentType.Slt }, + { "application/vnd.stepmania.stepchart", ContentType.Sm }, + { "application/vnd.stardivision.math", ContentType.Smf }, + { "application/smil+xml", ContentType.Smil }, + { "video/x-smv", ContentType.Smv }, + { "application/vnd.stepmania.package", ContentType.Smzip }, + { "application/x-font-snf", ContentType.Snf }, + { "text/spdx", ContentType.Spdx }, + { "application/vnd.yamaha.smaf-phrase", ContentType.Spf }, + { "application/x-futuresplash", ContentType.Spl }, + { "text/vnd.in3d.spot", ContentType.Spot }, + { "application/scvp-vp-response", ContentType.Spp }, + { "application/scvp-vp-request", ContentType.Spq }, + { "application/x-sql", ContentType.Sql }, + { "application/x-wais-source", ContentType.Src }, + { "application/x-subrip", ContentType.Srt }, + { "application/sru+xml", ContentType.Sru }, + { "application/sparql-results+xml", ContentType.Srx }, + { "application/ssdl+xml", ContentType.Ssdl }, + { "application/vnd.kodak-descriptor", ContentType.Sse }, + { "application/vnd.epson.ssf", ContentType.Ssf }, + { "application/ssml+xml", ContentType.Ssml }, + { "application/vnd.sailingtracker.track", ContentType.St }, + { "application/vnd.sun.xml.calc.template", ContentType.Stc }, + { "application/vnd.sun.xml.draw.template", ContentType.Std }, + { "application/vnd.wt.stf", ContentType.Stf }, + { "application/vnd.sun.xml.impress.template", ContentType.Sti }, + { "application/hyperstudio", ContentType.Stk }, + { "application/vnd.ms-pki.stl", ContentType.Stl }, + { "model/step+xml", ContentType.Stpx }, + { "model/step-xml+zip", ContentType.Stpxz }, + { "model/step+zip", ContentType.Stpz }, + { "application/vnd.pg.format", ContentType.Str }, + { "application/vnd.sun.xml.writer.template", ContentType.Stw }, + { "text/stylus", ContentType.Stylus }, + { "image/vnd.dvb.subtitle", ContentType.Sub }, + { "application/vnd.sus-calendar", ContentType.Sus }, + { "application/x-sv4cpio", ContentType.Sv4cpio }, + { "application/x-sv4crc", ContentType.Sv4crc }, + { "application/vnd.dvb.service", ContentType.Svc }, + { "application/vnd.svd", ContentType.Svd }, + { "image/svg+xml", ContentType.Svg }, + { "application/x-shockwave-flash", ContentType.Swf }, + { "application/vnd.aristanetworks.swi", ContentType.Swi }, + { "application/swid+xml", ContentType.Swidtag }, + { "application/vnd.sun.xml.calc", ContentType.Sxc }, + { "application/vnd.sun.xml.draw", ContentType.Sxd }, + { "application/vnd.sun.xml.writer.global", ContentType.Sxg }, + { "application/vnd.sun.xml.impress", ContentType.Sxi }, + { "application/vnd.sun.xml.math", ContentType.Sxm }, + { "application/vnd.sun.xml.writer", ContentType.Sxw }, + { "application/x-t3vm-image", ContentType.T3 }, + { "image/t38", ContentType.T38 }, + { "application/vnd.mynfc", ContentType.Taglet }, + { "application/vnd.tao.intent-module-archive", ContentType.Tao }, + { "image/vnd.tencent.tap", ContentType.Tap }, + { "application/x-tar", ContentType.Tar }, + { "application/vnd.3gpp2.tcap", ContentType.Tcap }, + { "application/x-tcl", ContentType.Tcl }, + { "application/urc-targetdesc+xml", ContentType.Td }, + { "application/vnd.smart.teacher", ContentType.Teacher }, + { "application/tei+xml", ContentType.Tei }, + { "application/x-tex", ContentType.Tex }, + { "application/x-texinfo", ContentType.Texinfo }, + { "text/plain", ContentType.Text }, + { "application/thraud+xml", ContentType.Tfi }, + { "application/x-tex-tfm", ContentType.Tfm }, + { "image/tiff-fx", ContentType.Tfx }, + { "image/x-tga", ContentType.Tga }, + { "application/vnd.ms-officetheme", ContentType.Thmx }, + { "image/tiff", ContentType.Tiff }, + { "application/vnd.tmobile-livetv", ContentType.Tmo }, + { "application/toml", ContentType.Toml }, + { "application/x-bittorrent", ContentType.Torrent }, + { "application/vnd.groove-tool-template", ContentType.Tpl }, + { "application/vnd.trid.tpt", ContentType.Tpt }, + { "application/vnd.trueapp", ContentType.Tra }, + { "application/trig", ContentType.Trig }, + { "application/x-msterminal", ContentType.Trm }, + { "video/mp2t", ContentType.Ts }, + { "application/timestamped-data", ContentType.Tsd }, + { "text/tab-separated-values", ContentType.Tsv }, + { "font/collection", ContentType.Ttc }, + { "font/ttf", ContentType.Ttf }, + { "text/turtle", ContentType.Ttl }, + { "application/ttml+xml", ContentType.Ttml }, + { "application/vnd.simtech-mindmapper", ContentType.Twd }, + { "application/vnd.genomatix.tuxedo", ContentType.Txd }, + { "application/vnd.mobius.txf", ContentType.Txf }, + { "application/x-authorware-bin", ContentType.U32 }, + { "message/global-delivery-status", ContentType.U8dsn }, + { "message/global-headers", ContentType.U8hdr }, + { "message/global-disposition-notification", ContentType.U8mdn }, + { "message/global", ContentType.U8msg }, + { "application/ubjson", ContentType.Ubj }, + { "application/x-debian-package", ContentType.Udeb }, + { "application/vnd.ufdl", ContentType.Ufdl }, + { "application/x-glulx", ContentType.Ulx }, + { "application/vnd.umajin", ContentType.Umj }, + { "application/vnd.unity", ContentType.Unityweb }, + { "application/vnd.uoml+xml", ContentType.Uoml }, + { "text/uri-list", ContentType.Uri }, + { "model/vnd.usdz+zip", ContentType.Usdz }, + { "application/x-ustar", ContentType.Ustar }, + { "application/vnd.uiq.theme", ContentType.Utz }, + { "text/x-uuencode", ContentType.Uu }, + { "audio/vnd.dece.audio", ContentType.Uva }, + { "application/vnd.dece.data", ContentType.Uvd }, + { "image/vnd.dece.graphic", ContentType.Uvg }, + { "video/vnd.dece.hd", ContentType.Uvh }, + { "video/vnd.dece.mobile", ContentType.Uvm }, + { "video/vnd.dece.pd", ContentType.Uvp }, + { "video/vnd.dece.sd", ContentType.Uvs }, + { "application/vnd.dece.ttml+xml", ContentType.Uvt }, + { "video/vnd.uvvu.mp4", ContentType.Uvu }, + { "video/vnd.dece.video", ContentType.Uvv }, + { "application/vnd.dece.zip", ContentType.Uvz }, + { "application/x-virtualbox-vbox", ContentType.Vbox }, + { "text/vcard", ContentType.Vcard }, + { "application/x-cdlink", ContentType.Vcd }, + { "text/x-vcard", ContentType.Vcf }, + { "application/vnd.groove-vcard", ContentType.Vcg }, + { "text/x-vcalendar", ContentType.Vcs }, + { "application/vnd.vcx", ContentType.Vcx }, + { "application/x-virtualbox-vdi", ContentType.Vdi }, + { "model/vnd.sap.vds", ContentType.Vds }, + { "application/x-virtualbox-vhd", ContentType.Vhd }, + { "application/vnd.visionary", ContentType.Vis }, + { "video/vnd.vivo", ContentType.Viv }, + { "application/x-virtualbox-vmdk", ContentType.Vmdk }, + { "video/x-ms-vob", ContentType.Vob }, + { "model/vrml", ContentType.Vrml }, + { "application/vnd.vsf", ContentType.Vsf }, + { "application/vnd.visio", ContentType.Vss }, + { "image/vnd.valve.source.texture", ContentType.Vtf }, + { "text/vtt", ContentType.Vtt }, + { "model/vnd.vtu", ContentType.Vtu }, + { "application/voicexml+xml", ContentType.Vxml }, + { "application/x-doom", ContentType.Wad }, + { "application/vnd.sun.wadl+xml", ContentType.Wadl }, + { "application/wasm", ContentType.Wasm }, + { "audio/wav", ContentType.Wav }, + { "audio/x-ms-wax", ContentType.Wax }, + { "image/vnd.wap.wbmp", ContentType.Wbmp }, + { "application/vnd.criticaltools.wbs+xml", ContentType.Wbs }, + { "application/vnd.wap.wbxml", ContentType.Wbxml }, + { "image/vnd.ms-photo", ContentType.Wdp }, + { "audio/webm", ContentType.Weba }, + { "application/x-web-app-manifest+json", ContentType.Webapp }, + { "video/webm", ContentType.Webm }, + { "image/webp", ContentType.Webp }, + { "application/vnd.pmi.widget", ContentType.Wg }, + { "application/widget", ContentType.Wgt }, + { "application/vnd.ms-works", ContentType.Wks }, + { "video/x-ms-wm", ContentType.Wm }, + { "audio/x-ms-wma", ContentType.Wma }, + { "application/x-ms-wmd", ContentType.Wmd }, + { "application/x-msmetafile", ContentType.Wmf }, + { "text/vnd.wap.wml", ContentType.Wml }, + { "application/vnd.wap.wmlc", ContentType.Wmlc }, + { "text/vnd.wap.wmlscript", ContentType.Wmls }, + { "application/vnd.wap.wmlscriptc", ContentType.Wmlsc }, + { "video/x-ms-wmv", ContentType.Wmv }, + { "video/x-ms-wmx", ContentType.Wmx }, + { "application/x-ms-wmz", ContentType.Wmz }, + { "font/woff", ContentType.Woff }, + { "font/woff2", ContentType.Woff2 }, + { "application/vnd.wordperfect", ContentType.Wpd }, + { "application/vnd.ms-wpl", ContentType.Wpl }, + { "application/vnd.wqd", ContentType.Wqd }, + { "application/x-mswrite", ContentType.Wri }, + { "message/vnd.wfa.wsc", ContentType.Wsc }, + { "application/wsdl+xml", ContentType.Wsdl }, + { "application/wspolicy+xml", ContentType.Wspolicy }, + { "application/vnd.webturbo", ContentType.Wtb }, + { "video/x-ms-wvx", ContentType.Wvx }, + { "model/x3d+xml", ContentType.X3d }, + { "model/x3d+binary", ContentType.X3db }, + { "model/x3d+vrml", ContentType.X3dv }, + { "application/xaml+xml", ContentType.Xaml }, + { "application/x-silverlight-app", ContentType.Xap }, + { "application/vnd.xara", ContentType.Xar }, + { "application/xcap-att+xml", ContentType.Xav }, + { "application/x-ms-xbap", ContentType.Xbap }, + { "application/vnd.fujixerox.docuworks.binder", ContentType.Xbd }, + { "image/x-xbitmap", ContentType.Xbm }, + { "application/xcap-caps+xml", ContentType.Xca }, + { "application/calendar+xml", ContentType.Xcs }, + { "application/xcap-diff+xml", ContentType.Xdf }, + { "application/vnd.syncml.dm+xml", ContentType.Xdm }, + { "application/vnd.adobe.xdp+xml", ContentType.Xdp }, + { "application/dssc+xml", ContentType.Xdssc }, + { "application/vnd.fujixerox.docuworks", ContentType.Xdw }, + { "application/xcap-el+xml", ContentType.Xel }, + { "application/xenc+xml", ContentType.Xenc }, + { "application/patch-ops-error+xml", ContentType.Xer }, + { "application/vnd.adobe.xfdf", ContentType.Xfdf }, + { "application/vnd.xfdl", ContentType.Xfdl }, + { "application/xhtml+xml", ContentType.Xhtml }, + { "image/vnd.xiff", ContentType.Xif }, + { "application/vnd.ms-excel.addin.macroenabled.12", ContentType.Xlam }, + { "application/x-xliff+xml", ContentType.Xlf }, + { "application/vnd.ms-excel", ContentType.Xls }, + { "application/vnd.ms-excel.sheet.binary.macroenabled.12", ContentType.Xlsb }, + { "application/vnd.ms-excel.sheet.macroenabled.12", ContentType.Xlsm }, + { "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", ContentType.Xlsx }, + { "application/vnd.ms-excel.template.macroenabled.12", ContentType.Xltm }, + { "application/vnd.openxmlformats-officedocument.spreadsheetml.template", ContentType.Xltx }, + { "audio/xm", ContentType.Xm }, + { "application/xml", ContentType.Xml }, + { "application/xcap-ns+xml", ContentType.Xns }, + { "application/vnd.olpc-sugar", ContentType.Xo }, + { "application/xop+xml", ContentType.Xop }, + { "application/x-xpinstall", ContentType.Xpi }, + { "application/xproc+xml", ContentType.Xpl }, + { "image/x-xpixmap", ContentType.Xpm }, + { "application/vnd.is-xpr", ContentType.Xpr }, + { "application/vnd.ms-xpsdocument", ContentType.Xps }, + { "application/vnd.intercon.formnet", ContentType.Xpw }, + { "application/xslt+xml", ContentType.Xslt }, + { "application/vnd.syncml+xml", ContentType.Xsm }, + { "application/vnd.mozilla.xul+xml", ContentType.Xul }, + { "application/xv+xml", ContentType.Xvml }, + { "image/x-xwindowdump", ContentType.Xwd }, + { "chemical/x-xyz", ContentType.Xyz }, + { "application/x-xz", ContentType.Xz }, + { "text/yaml", ContentType.Yaml }, + { "application/yang", ContentType.Yang }, + { "application/yin+xml", ContentType.Yin }, + { "text/x-suse-ymp", ContentType.Ymp }, + { "application/x-zmachine", ContentType.Z1 }, + { "application/vnd.zzazz.deck+xml", ContentType.Zaz }, + { "application/zip", ContentType.Zip }, + { "application/vnd.zul", ContentType.Zir }, + { "application/vnd.handheld-entertainment+xml", ContentType.Zmm }, + }; + } +} diff --git a/Net.Http/src/Helpers/TransportReader.cs b/Net.Http/src/Helpers/TransportReader.cs new file mode 100644 index 0000000..a37bfe9 --- /dev/null +++ b/Net.Http/src/Helpers/TransportReader.cs @@ -0,0 +1,114 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: TransportReader.cs +* +* TransportReader.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; + +using VNLib.Utils; +using VNLib.Utils.IO; + + +namespace VNLib.Net.Http.Core +{ + + /// <summary> + /// Structure implementation of <see cref="IVnTextReader"/> + /// </summary> + internal struct TransportReader : IVnTextReader + { + ///<inheritdoc/> + public readonly Encoding Encoding => _encoding; + ///<inheritdoc/> + public readonly ReadOnlyMemory<byte> LineTermination => _lineTermination; + ///<inheritdoc/> + public readonly Stream BaseStream => _transport; + + + private readonly SharedHeaderReaderBuffer BinBuffer; + private readonly Encoding _encoding; + private readonly Stream _transport; + private readonly ReadOnlyMemory<byte> _lineTermination; + + private int BufWindowStart; + private int BufWindowEnd; + + /// <summary> + /// Initializes a new <see cref="TransportReader"/> for reading text lines from the transport stream + /// </summary> + /// <param name="transport">The transport stream to read data from</param> + /// <param name="buffer">The shared binary buffer</param> + /// <param name="encoding">The encoding to use when reading bianry</param> + /// <param name="lineTermination">The line delimiter to search for</param> + public TransportReader(Stream transport, SharedHeaderReaderBuffer buffer, Encoding encoding, ReadOnlyMemory<byte> lineTermination) + { + BufWindowEnd = 0; + BufWindowStart = 0; + _encoding = encoding; + _transport = transport; + _lineTermination = lineTermination; + BinBuffer = buffer; + } + + ///<inheritdoc/> + public readonly int Available => BufWindowEnd - BufWindowStart; + + ///<inheritdoc/> + public readonly Span<byte> BufferedDataWindow => BinBuffer.BinBuffer[BufWindowStart..BufWindowEnd]; + + + ///<inheritdoc/> + public void Advance(int count) => BufWindowStart += count; + ///<inheritdoc/> + public void FillBuffer() + { + //Get a buffer from the end of the current window to the end of the buffer + Span<byte> bufferWindow = BinBuffer.BinBuffer[BufWindowEnd..]; + //Read from stream + int read = _transport.Read(bufferWindow); + //Update the end of the buffer window to the end of the read data + BufWindowEnd += read; + } + ///<inheritdoc/> + public ERRNO CompactBufferWindow() + { + //No data to compact if window is not shifted away from start + if (BufWindowStart > 0) + { + //Get span over engire buffer + Span<byte> buffer = BinBuffer.BinBuffer; + //Get data within window + Span<byte> usedData = buffer[BufWindowStart..BufWindowEnd]; + //Copy remaining to the begining of the buffer + usedData.CopyTo(buffer); + //Buffer window start is 0 + BufWindowStart = 0; + //Buffer window end is now the remaining size + BufWindowEnd = usedData.Length; + } + //Return the number of bytes of available space + return BinBuffer.BinLength - BufWindowEnd; + } + } +} diff --git a/Net.Http/src/Helpers/VnWebHeaderCollection.cs b/Net.Http/src/Helpers/VnWebHeaderCollection.cs new file mode 100644 index 0000000..f67e7db --- /dev/null +++ b/Net.Http/src/Helpers/VnWebHeaderCollection.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: VnWebHeaderCollection.cs +* +* VnWebHeaderCollection.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Collections.Generic; + + +namespace VNLib.Net.Http +{ + ///<inheritdoc/> + public sealed class VnWebHeaderCollection : WebHeaderCollection, IEnumerable<KeyValuePair<string?, string?>> + { + IEnumerator<KeyValuePair<string?, string?>> IEnumerable<KeyValuePair<string?, string?>>.GetEnumerator() + { + for (int i = 0; i < Keys.Count; i++) + { + yield return new(Keys[i], Get(i)); + } + } + } +} diff --git a/Net.Http/src/Helpers/WebHeaderExtensions.cs b/Net.Http/src/Helpers/WebHeaderExtensions.cs new file mode 100644 index 0000000..e51bdd5 --- /dev/null +++ b/Net.Http/src/Helpers/WebHeaderExtensions.cs @@ -0,0 +1,60 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: WebHeaderExtensions.cs +* +* WebHeaderExtensions.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Net; +using System.Runtime.CompilerServices; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Extends the <see cref="WebHeaderCollection"/> to provide some check methods + /// </summary> + public static class WebHeaderExtensions + { + /// <summary> + /// Determines if the specified request header has been set in the current header collection + /// </summary> + /// <param name="headers"></param> + /// <param name="header">Header value to check</param> + /// <returns>true if the header was set, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HeaderSet(this WebHeaderCollection headers, HttpRequestHeader header) => !string.IsNullOrWhiteSpace(headers[header]); + /// <summary> + /// Determines if the specified response header has been set in the current header collection + /// </summary> + /// <param name="headers"></param> + /// <param name="header">Header value to check</param> + /// <returns>true if the header was set, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HeaderSet(this WebHeaderCollection headers, HttpResponseHeader header) => !string.IsNullOrWhiteSpace(headers[header]); + /// <summary> + /// Determines if the specified header has been set in the current header collection + /// </summary> + /// <param name="headers"></param> + /// <param name="header">Header value to check</param> + /// <returns>true if the header was set, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HeaderSet(this WebHeaderCollection headers, string header) => !string.IsNullOrWhiteSpace(headers[header]); + } +}
\ No newline at end of file diff --git a/Net.Http/src/HttpConfig.cs b/Net.Http/src/HttpConfig.cs new file mode 100644 index 0000000..8e73176 --- /dev/null +++ b/Net.Http/src/HttpConfig.cs @@ -0,0 +1,154 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: HttpConfig.cs +* +* HttpConfig.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.IO.Compression; + +using VNLib.Utils.Logging; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Represents configration variables used to create the instance and manage http connections + /// </summary> + public readonly struct HttpConfig + { + public HttpConfig(ILogProvider log) + { + ConnectionKeepAlive = TimeSpan.FromSeconds(100); + ServerLog = log; + } + + /// <summary> + /// A log provider that all server related log entiries will be written to + /// </summary> + public readonly ILogProvider ServerLog { get; } + /// <summary> + /// The absolute request entity body size limit in bytes + /// </summary> + public readonly int MaxUploadSize { get; init; } = 5 * 1000 * 1024; + /// <summary> + /// The maximum size in bytes allowed for an MIME form-data content type upload + /// </summary> + /// <remarks>Set to 0 to disabled mulit-part/form-data uploads</remarks> + public readonly int MaxFormDataUploadSize { get; init; } = 40 * 1024; + /// <summary> + /// The maximum buffer size to use when parsing Multi-part/Form-data file uploads + /// </summary> + /// <remarks> + /// This value is used to create the buffer used to read data from the input stream + /// into memory for parsing. Form-data uploads must be parsed in memory because + /// the data is not delimited by a content length. + /// </remarks> + public readonly int FormDataBufferSize { get; init; } = 8192; + /// <summary> + /// The maximum response entity size in bytes for which the library will allow compresssing response data + /// </summary> + /// <remarks>Set this value to 0 to disable response compression</remarks> + public readonly int CompressionLimit { get; init; } = 1000 * 1024; + /// <summary> + /// The minimum size (in bytes) of respones data that will be compressed + /// </summary> + public readonly int CompressionMinimum { get; init; } = 4096; + /// <summary> + /// The maximum amount of time to listen for data from a connected, but inactive transport connection + /// before closing them + /// </summary> + public readonly TimeSpan ConnectionKeepAlive { get; init; } + /// <summary> + /// The encoding to use when sending and receiving HTTP data + /// </summary> + public readonly Encoding HttpEncoding { get; init; } = Encoding.UTF8; + /// <summary> + /// Sets the compression level for response entity streams of all supported types when + /// compression is used. + /// </summary> + public readonly CompressionLevel CompressionLevel { get; init; } = CompressionLevel.Optimal; + /// <summary> + /// Sets the default Http version for responses when the client version cannot be parsed from the request + /// </summary> + public readonly HttpVersion DefaultHttpVersion { get; init; } = HttpVersion.Http11; + /// <summary> + /// The buffer size used to read HTTP headers from the transport. + /// </summary> + /// <remarks> + /// Setting this value too low will result in header parsing failures + /// and 400 Bad Request responses. Setting it too high can result in + /// resource abuse or high memory usage. 8k is usually a good value. + /// </remarks> + public readonly int HeaderBufferSize { get; init; } = 8192; + /// <summary> + /// The amount of time (in milliseconds) to wait for data on a connection that is in a receive + /// state, aka active receive. + /// </summary> + public readonly int ActiveConnectionRecvTimeout { get; init; } = 5000; + /// <summary> + /// The amount of time (in milliseconds) to wait for data to be send to the client before + /// the connection is closed + /// </summary> + public readonly int SendTimeout { get; init; } = 5000; + /// <summary> + /// The maximum number of request headers allowed per request + /// </summary> + public readonly int MaxRequestHeaderCount { get; init; } = 100; + /// <summary> + /// The maximum number of open transport connections, before 503 errors + /// will be returned and new connections closed. + /// </summary> + /// <remarks>Set to 0 to disable request processing. Causes perminant 503 results</remarks> + public readonly int MaxOpenConnections { get; init; } = int.MaxValue; + /// <summary> + /// The size (in bytes) of the http response header accumulator buffer. + /// </summary> + /// <remarks> + /// Http responses use an internal accumulator to buffer all response headers + /// before writing them to the transport in on write operation. If this value + /// is too low, the response will fail to write. If it is too high, it + /// may cause resource exhaustion or high memory usage. + /// </remarks> + public readonly int ResponseHeaderBufferSize { get; init; } = 16 * 1024; + /// <summary> + /// The size (in bytes) of the buffer to use to discard unread request entity bodies + /// </summary> + public readonly int DiscardBufferSize { get; init; } = 64 * 1024; + /// <summary> + /// The size of the buffer to use when writing response data to the transport + /// </summary> + /// <remarks> + /// This value is the size of the buffer used to copy data from the response + /// entity stream, to the transport stream. + /// </remarks> + public readonly int ResponseBufferSize { get; init; } = 32 * 1024; + /// <summary> + /// The size of the buffer used to accumulate chunked response data before writing to the transport + /// </summary> + public readonly int ChunkedResponseAccumulatorSize { get; init; } = 64 * 1024; + /// <summary> + /// An <see cref="ILogProvider"/> for writing verbose request logs. Set to <c>null</c> + /// to disable verbose request logging + /// </summary> + public readonly ILogProvider? RequestDebugLog { get; init; } = null; + } +}
\ No newline at end of file diff --git a/Net.Http/src/IAlternateProtocol.cs b/Net.Http/src/IAlternateProtocol.cs new file mode 100644 index 0000000..dc7072b --- /dev/null +++ b/Net.Http/src/IAlternateProtocol.cs @@ -0,0 +1,46 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IAlternateProtocol.cs +* +* IAlternateProtocol.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Allows implementation for a protocol swtich from HTTP to another protocol + /// </summary> + public interface IAlternateProtocol + { + /// <summary> + /// Initializes and executes the protocol-switch and the protocol handler + /// that is stored + /// </summary> + /// <param name="transport">The prepared transport stream for the new protocol</param> + /// <param name="handlerToken">A cancelation token that the caller may pass for operation cancelation and cleanup</param> + /// <returns>A task that will be awaited by the server, that when complete, will cleanup resources held by the connection</returns> + Task RunAsync(Stream transport, CancellationToken handlerToken); + } +}
\ No newline at end of file diff --git a/Net.Http/src/IConnectionInfo.cs b/Net.Http/src/IConnectionInfo.cs new file mode 100644 index 0000000..17ee16f --- /dev/null +++ b/Net.Http/src/IConnectionInfo.cs @@ -0,0 +1,150 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IConnectionInfo.cs +* +* IConnectionInfo.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Text; +using System.Collections.Generic; +using System.Security.Authentication; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Represents a client's connection info as interpreted by the current server + /// </summary> + /// <remarks>Methods and properties are undefined when <see cref="IWebRoot.ClientConnectedAsync(IHttpEvent)"/> returns</remarks> + public interface IConnectionInfo + { + /// <summary> + /// Full request uri of current connection + /// </summary> + Uri RequestUri { get; } + /// <summary> + /// Current request path. Shortcut to <seealso cref="RequestUri"/> <see cref="Uri.LocalPath"/> + /// </summary> + string Path => RequestUri.LocalPath; + /// <summary> + /// Current connection's user-agent header, (may be null if no user-agent header found) + /// </summary> + string? UserAgent { get; } + /// <summary> + /// Current connection's headers + /// </summary> + IHeaderCollection Headers { get; } + /// <summary> + /// A value that indicates if the connection's origin header was set and it's + /// authority segment does not match the <see cref="RequestUri"/> authority + /// segment. + /// </summary> + bool CrossOrigin { get; } + /// <summary> + /// Is the current connecion a websocket request + /// </summary> + bool IsWebSocketRequest { get; } + /// <summary> + /// Request specified content-type + /// </summary> + ContentType ContentType { get; } + /// <summary> + /// Current request's method + /// </summary> + HttpMethod Method { get; } + /// <summary> + /// The current connection's HTTP protocol version + /// </summary> + HttpVersion ProtocolVersion { get; } + /// <summary> + /// Is the connection using transport security? + /// </summary> + bool IsSecure { get; } + /// <summary> + /// The negotiated transport protocol for the current connection + /// </summary> + SslProtocols SecurityProtocol { get; } + /// <summary> + /// Origin header of current connection if specified, null otherwise + /// </summary> + Uri? Origin { get; } + /// <summary> + /// Referer header of current connection if specified, null otherwise + /// </summary> + Uri? Referer { get; } + /// <summary> + /// The parsed range header, or -1,-1 if the range header was not set + /// </summary> + Tuple<long, long>? Range { get; } + /// <summary> + /// The server endpoint that accepted the connection + /// </summary> + IPEndPoint LocalEndpoint { get; } + /// <summary> + /// The raw <see cref="IPEndPoint"/> of the downstream connection. + /// </summary> + IPEndPoint RemoteEndpoint { get; } + /// <summary> + /// The encoding type used to decode and encode character data to and from the current client + /// </summary> + Encoding Encoding { get; } + /// <summary> + /// A <see cref="IReadOnlyDictionary{TKey, TValue}"/> of client request cookies + /// </summary> + IReadOnlyDictionary<string, string> RequestCookies { get; } + /// <summary> + /// Gets an <see cref="IEnumerator{T}"/> for the parsed accept header values + /// </summary> + IEnumerable<string> Accept { get; } + /// <summary> + /// Gets the underlying transport security information for the current connection + /// </summary> + TransportSecurityInfo? TransportSecurity { get; } + + /// <summary> + /// Determines if the client accepts the response content type + /// </summary> + /// <param name="type">The desired content type</param> + /// <returns>True if the client accepts the content type, false otherwise</returns> + bool Accepts(ContentType type); + + /// <summary> + /// Determines if the client accepts the response content type + /// </summary> + /// <param name="contentType">The desired content type</param> + /// <returns>True if the client accepts the content type, false otherwise</returns> + bool Accepts(string contentType); + + /// <summary> + /// Adds a new cookie to the response. If a cookie with the same name and value + /// has been set, the old cookie is replaced with the new one. + /// </summary> + /// <param name="name">Cookie name/id</param> + /// <param name="value">Value to be stored in cookie</param> + /// <param name="domain">Domain for cookie to operate</param> + /// <param name="path">Path to store cookie</param> + /// <param name="Expires">Timespan representing how long the cookie should exist</param> + /// <param name="sameSite">Samesite attribute, Default = Lax</param> + /// <param name="httpOnly">Specify the HttpOnly flag</param> + /// <param name="secure">Specify the Secure flag</param> + void SetCookie(string name, string value, string? domain, string? path, TimeSpan Expires, CookieSameSite sameSite, bool httpOnly, bool secure); + } +}
\ No newline at end of file diff --git a/Net.Http/src/IHeaderCollection.cs b/Net.Http/src/IHeaderCollection.cs new file mode 100644 index 0000000..f7f147a --- /dev/null +++ b/Net.Http/src/IHeaderCollection.cs @@ -0,0 +1,85 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IHeaderCollection.cs +* +* IHeaderCollection.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Net; +using System.Collections.Generic; + +namespace VNLib.Net.Http +{ + /// <summary> + /// The container for request and response headers + /// </summary> + public interface IHeaderCollection + { + /// <summary> + /// Allows for enumeratring all requesest headers + /// </summary> + IEnumerable<KeyValuePair<string, string>> RequestHeaders { get; } + /// <summary> + /// Allows for enumeratring all response headers + /// </summary> + IEnumerable<KeyValuePair<string, string>> ResponseHeaders { get; } + /// <summary> + /// Gets request header, or sets a response header + /// </summary> + /// <param name="index"></param> + /// <returns>Request header with key</returns> + string? this[string index] { get; set; } + /// <summary> + /// Sets a response header only with a response header index + /// </summary> + /// <param name="index">Response header</param> + string this[HttpResponseHeader index] { set; } + /// <summary> + /// Gets a request header + /// </summary> + /// <param name="index">The request header enum </param> + string? this[HttpRequestHeader index] { get; } + /// <summary> + /// Determines if the given header is set in current response headers + /// </summary> + /// <param name="header">Header value to check response headers for</param> + /// <returns>true if header exists in current response headers, false otherwise</returns> + bool HeaderSet(HttpResponseHeader header); + /// <summary> + /// Determines if the given request header is set in current request headers + /// </summary> + /// <param name="header">Header value to check request headers for</param> + /// <returns>true if header exists in current request headers, false otherwise</returns> + bool HeaderSet(HttpRequestHeader header); + + /// <summary> + /// Overwrites (sets) the given response header to the exact value specified + /// </summary> + /// <param name="header">The enumrated header id</param> + /// <param name="value">The value to specify</param> + void Append(HttpResponseHeader header, string? value); + /// <summary> + /// Overwrites (sets) the given response header to the exact value specified + /// </summary> + /// <param name="header">The header name</param> + /// <param name="value">The value to specify</param> + void Append(string header, string? value); + } +}
\ No newline at end of file diff --git a/Net.Http/src/IMemoryResponseEntity.cs b/Net.Http/src/IMemoryResponseEntity.cs new file mode 100644 index 0000000..aa77f58 --- /dev/null +++ b/Net.Http/src/IMemoryResponseEntity.cs @@ -0,0 +1,69 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IMemoryResponseEntity.cs +* +* IMemoryResponseEntity.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Net.Http +{ + /// <summary> + /// <para> + /// A forward only memory backed response entity body reader. This interface exists + /// to provide a memory-backed response body that will be written "directly" to the + /// response stream. This avoids a buffer allocation and a copy. + /// </para> + /// <para> + /// The entity is only read foward, one time, so it is not seekable. + /// </para> + /// <para> + /// The <see cref="Close"/> method is always called by internal lifecycle hooks + /// when the entity is no longer needed. <see cref="Close"/> should avoid raising + /// excptions. + /// </para> + /// </summary> + public interface IMemoryResponseReader + { + /// <summary> + /// Gets a readonly buffer containing the remaining + /// data to be written + /// </summary> + /// <returns>A memory segment to send to the client</returns> + ReadOnlyMemory<byte> GetMemory(); + + /// <summary> + /// Advances the buffer by the number of bytes written + /// </summary> + /// <param name="written">The number of bytes written</param> + void Advance(int written); + + /// <summary> + /// The number of bytes remaining to send + /// </summary> + int Remaining { get; } + + /// <summary> + /// Raised when reading has completed + /// </summary> + void Close(); + } +}
\ No newline at end of file diff --git a/Net.Http/src/ITransportContext.cs b/Net.Http/src/ITransportContext.cs new file mode 100644 index 0000000..bd6ce05 --- /dev/null +++ b/Net.Http/src/ITransportContext.cs @@ -0,0 +1,71 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ITransportContext.cs +* +* ITransportContext.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.IO; +using System.Net; +using System.Threading.Tasks; +using System.Security.Authentication; + + +namespace VNLib.Net.Http +{ + /// <summary> + /// Represents an active connection for application data processing + /// </summary> + public interface ITransportContext + { + /// <summary> + /// The transport network stream for application data marshaling + /// </summary> + Stream ConnectionStream { get; } + /// <summary> + /// The transport security layer security protocol + /// </summary> + SslProtocols SslVersion { get; } + /// <summary> + /// A copy of the local endpoint of the listening socket + /// </summary> + IPEndPoint LocalEndPoint { get; } + /// <summary> + /// The <see cref="IPEndPoint"/> representing the client's connection information + /// </summary> + IPEndPoint RemoteEndpoint { get; } + + /// <summary> + /// Closes the connection when its no longer in use and cleans up held resources. + /// </summary> + /// <returns></returns> + /// <remarks> + /// This method will always be called by the server when a connection is complete + /// regardless of the state of the trasnport + /// </remarks> + ValueTask CloseConnectionAsync(); + + /// <summary> + /// Attemts to get the transport security details for the connection + /// </summary> + /// <returns>A the <see cref="TransportSecurityInfo"/> structure if applicable, null otherwise</returns> + TransportSecurityInfo? GetSecurityInfo(); + } +}
\ No newline at end of file diff --git a/Net.Http/src/ITransportProvider.cs b/Net.Http/src/ITransportProvider.cs new file mode 100644 index 0000000..13aae57 --- /dev/null +++ b/Net.Http/src/ITransportProvider.cs @@ -0,0 +1,51 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: ITransportProvider.cs +* +* ITransportProvider.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Http +{ + /// <summary> + /// Listens for network connections and captures the information + /// required for application processing + /// </summary> + public interface ITransportProvider + { + /// <summary> + /// Begins listening for connections (binds a socket if necessary) and is + /// called before the server begins listening for connections. + /// </summary> + /// <param name="stopToken">A token that is cancelled when the server is closed</param> + void Start(CancellationToken stopToken); + + /// <summary> + /// Waits for a new connection to be established and returns its context. This method + /// should only return an established connection (ie: connected socket). + /// </summary> + /// <param name="cancellation">A token to cancel the wait operation</param> + /// <returns>A <see cref="ValueTask"/> that returns an established connection</returns> + ValueTask<ITransportContext> AcceptAsync(CancellationToken cancellation); + } +}
\ No newline at end of file diff --git a/Net.Http/src/IWebRoot.cs b/Net.Http/src/IWebRoot.cs new file mode 100644 index 0000000..9b1a9c8 --- /dev/null +++ b/Net.Http/src/IWebRoot.cs @@ -0,0 +1,58 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: IWebRoot.cs +* +* IWebRoot.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading.Tasks; +using System.Collections.Generic; + + +namespace VNLib.Net.Http +{ + /// <summary> + /// Represents a root identifying the main endpoints of the server, and the primary processing actions + /// for requests to this endpoint + /// </summary> + public interface IWebRoot + { + /// <summary> + /// The hostname the server will listen for, and the hostname that will identify this root when a connection requests it + /// </summary> + string Hostname { get; } + /// <summary> + /// <para> + /// The main event handler for user code to process a request + /// </para> + /// <para> + /// NOTE: This function must be thread-safe! + /// </para> + /// </summary> + /// <param name="httpEvent">An active, unprocessed event capturing the request infomration into a standard format</param> + /// <returns>A <see cref="ValueTask"/> that the processor will await until the entity has been processed</returns> + ValueTask ClientConnectedAsync(IHttpEvent httpEvent); + /// <summary> + /// "Low-Level" 301 redirects + /// </summary> + IReadOnlyDictionary<string, Redirect> Redirects { get; } + } +}
\ No newline at end of file diff --git a/Net.Http/src/TransportSecurityInfo.cs b/Net.Http/src/TransportSecurityInfo.cs new file mode 100644 index 0000000..7c7a79c --- /dev/null +++ b/Net.Http/src/TransportSecurityInfo.cs @@ -0,0 +1,121 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Http +* File: TransportSecurityInfo.cs +* +* TransportSecurityInfo.cs is part of VNLib.Net.Http which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Http is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Http is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Net.Security; +using System.Security.Authentication; +using System.Security.Cryptography.X509Certificates; + + +namespace VNLib.Net.Http +{ + + /// <summary> + /// Gets the transport TLS security information for the current connection + /// </summary> + public readonly struct TransportSecurityInfo + { + /// <summary> + /// Gets a Boolean value that indicates whether the certificate revocation list is checked during the certificate validation process. + /// </summary> + /// <returns>true if the certificate revocation list is checked during validation; otherwise, false.</returns> + public readonly bool CheckCertRevocationStatus { get; init; } + + /// <summary> + /// Gets a value that identifies the bulk encryption algorithm used by the connection. + /// </summary> + public readonly CipherAlgorithmType CipherAlgorithm { get; init; } + + /// <summary> + /// Gets a value that identifies the strength of the cipher algorithm used by the connection. + /// </summary> + public readonly int CipherStrength { get; init; } + + /// <summary> + /// Gets the algorithm used for generating message authentication codes (MACs). + /// </summary> + public readonly HashAlgorithmType HashAlgorithm { get; init; } + + /// <summary> + /// Gets a value that identifies the strength of the hash algorithm used by this instance. + /// </summary> + public readonly int HashStrength { get; init; } + + /// <summary> + /// Gets a Boolean value that indicates whether authentication was successful. + /// </summary> + public readonly bool IsAuthenticated { get; init; } + + /// <summary> + /// Gets a Boolean value that indicates whether this connection uses data encryption. + /// </summary> + public readonly bool IsEncrypted { get; init; } + + /// <summary> + /// Gets a Boolean value that indicates whether both server and client have been authenticated. + /// </summary> + public readonly bool IsMutuallyAuthenticated { get; init; } + + /// <summary> + /// Gets a Boolean value that indicates whether the data sent using this connection is signed. + /// </summary> + public readonly bool IsSigned { get; init; } + + /// <summary> + /// Gets the key exchange algorithm used by this connection + /// </summary> + public readonly ExchangeAlgorithmType KeyExchangeAlgorithm { get; init; } + + /// <summary> + /// Gets a value that identifies the strength of the key exchange algorithm used by the transport connection + /// </summary> + public readonly int KeyExchangeStrength { get; init; } + + /// <summary> + /// Gets the certificate used to authenticate the local endpoint. + /// </summary> + public readonly X509Certificate? LocalCertificate { get; init; } + + /// <summary> + /// The negotiated application protocol in TLS handshake. + /// </summary> + public readonly SslApplicationProtocol NegotiatedApplicationProtocol { get; init; } + + /// <summary> + /// Gets the cipher suite which was negotiated for this connection. + /// </summary> + public readonly TlsCipherSuite NegotiatedCipherSuite { get; init; } + + /// <summary> + /// Gets the certificate used to authenticate the remote endpoint. + /// </summary> + public readonly X509Certificate? RemoteCertificate { get; init; } + + /// <summary> + /// Gets the TransportContext used for authentication using extended protection. + /// </summary> + public readonly TransportContext TransportContext { get; init; } + } +} diff --git a/Net.Http/src/VNLib.Net.Http.csproj b/Net.Http/src/VNLib.Net.Http.csproj new file mode 100644 index 0000000..30e698c --- /dev/null +++ b/Net.Http/src/VNLib.Net.Http.csproj @@ -0,0 +1,57 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <RootNamespace>VNLib.Net.Http</RootNamespace> + <Authors>Vaughn Nugent</Authors> + <Company>$(Authors)</Company> + <Product>VNLib HTTP Library</Product> + <Description>Provides a high performance HTTP 0.9-1.1 application processing layer for handling transport *agnostic connections and asynchronous event support for applications serving HTTP +requests such as web content. This library has a large focus on low/no GC allocations using unmanaged memory support provided by the VNLib.Utils library. No external dependencies +outside of the VNLib ecosystem are required. The VNLib.Plugins and VNLib.Plugins.Essentials libraries are highly recommended for serving web content.</Description> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <PackageId>VNLib.Net.Http</PackageId> + <Version>1.0.1.5</Version> + <NeutralLanguage>en-US</NeutralLanguage> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <AssemblyName>VNLib.Net.Http</AssemblyName> + <Nullable>enable</Nullable> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <AnalysisLevel>latest-all</AnalysisLevel> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <!-- Resolve nuget dll files and store them in the output dir --> + <PropertyGroup> + <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)'=='Debug'"> + <CheckForOverflowUnderflow>True</CheckForOverflowUnderflow> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + </ItemGroup> + +</Project> diff --git a/Net.Messaging.FBM/LICENSE.txt b/Net.Messaging.FBM/LICENSE.txt new file mode 100644 index 0000000..147bcd6 --- /dev/null +++ b/Net.Messaging.FBM/LICENSE.txt @@ -0,0 +1,195 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU Affero GPL version 3.0 (or any later version). + +GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/> +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. +Preamble + +The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. + +A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. + +The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. + +An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. + +The precise terms and conditions for copying, distribution and modification follow. +TERMS AND CONDITIONS +0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public License. + +"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based on the Program. + +To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. + +An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. +1. Source Code. + +The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. + +A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same work. +2. Basic Permissions. + +All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. +3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. + +When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. +4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. +5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified it, and giving a relevant date. + b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". + c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. + d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. + +A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. +6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: + + a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. + b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. + c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. + d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. + e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. + +"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. + +If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). + +The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. + +Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. +7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or + b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or + c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or + d) Limiting the use for publicity purposes of names of licensors or authors of the material; or + e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or + f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. + +All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. +8. Termination. + +You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). + +However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. + +Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. + +Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. +9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. +10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. +11. Patents. + +A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. + +If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. + +A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. +12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. +13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. +14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. + +Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. +15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. +16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Net.Messaging.FBM/README.md b/Net.Messaging.FBM/README.md new file mode 100644 index 0000000..aabed75 --- /dev/null +++ b/Net.Messaging.FBM/README.md @@ -0,0 +1,3 @@ +# VNLib.Net.Messaging.FBM + +High performance structured web-socket based asynchronous request/response messaging library for .NET.
\ No newline at end of file diff --git a/Net.Messaging.FBM/src/Client/ClientExtensions.cs b/Net.Messaging.FBM/src/Client/ClientExtensions.cs new file mode 100644 index 0000000..102b6c9 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/ClientExtensions.cs @@ -0,0 +1,69 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: ClientExtensions.cs +* +* ClientExtensions.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.CompilerServices; + +namespace VNLib.Net.Messaging.FBM.Client +{ + + public static class ClientExtensions + { + /// <summary> + /// Writes the location header of the requested resource + /// </summary> + /// <param name="request"></param> + /// <param name="location">The location address</param> + /// <exception cref="OutOfMemoryException"></exception> + public static void WriteLocation(this FBMRequest request, ReadOnlySpan<char> location) + { + request.WriteHeader(HeaderCommand.Location, location); + } + + /// <summary> + /// Writes the location header of the requested resource + /// </summary> + /// <param name="request"></param> + /// <param name="location">The location address</param> + /// <exception cref="OutOfMemoryException"></exception> + public static void WriteLocation(this FBMRequest request, Uri location) + { + request.WriteHeader(HeaderCommand.Location, location.ToString()); + } + + /// <summary> + /// If the <see cref="FBMResponse.IsSet"/> property is false, raises an <see cref="InvalidResponseException"/> + /// </summary> + /// <param name="response"></param> + /// <exception cref="InvalidResponseException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfNotSet(this in FBMResponse response) + { + if (!response.IsSet) + { + throw new InvalidResponseException("The response state is undefined (no response received)"); + } + } + } +} diff --git a/Net.Messaging.FBM/src/Client/FBMClient.cs b/Net.Messaging.FBM/src/Client/FBMClient.cs new file mode 100644 index 0000000..5353087 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/FBMClient.cs @@ -0,0 +1,475 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMClient.cs +* +* FBMClient.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Threading; +using System.Net.WebSockets; +using System.Threading.Tasks; +using System.Collections.Generic; +using System.Collections.Concurrent; + +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Logging; +using VNLib.Utils.Extensions; +using VNLib.Utils.Memory.Caching; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// A Fixed Buffer Message Protocol client. Allows for high performance client-server messaging + /// with minimal memory overhead. + /// </summary> + public class FBMClient : VnDisposeable, IStatefulConnection, ICacheHolder + { + /// <summary> + /// The WS connection query arguments to specify a receive buffer size + /// </summary> + public const string REQ_RECV_BUF_QUERY_ARG = "b"; + /// <summary> + /// The WS connection query argument to suggest a maximum response header buffer size + /// </summary> + public const string REQ_HEAD_BUF_QUERY_ARG = "hb"; + /// <summary> + /// The WS connection query argument to suggest a maximum message size + /// </summary> + public const string REQ_MAX_MESS_QUERY_ARG = "mx"; + + /// <summary> + /// Raised when the websocket has been closed because an error occured. + /// You may inspect the event args to determine the cause of the error. + /// </summary> + public event EventHandler<FMBClientErrorEventArgs>? ConnectionClosedOnError; + /// <summary> + /// Raised when the client listener operaiton has completed as a normal closure + /// </summary> + public event EventHandler? ConnectionClosed; + + private readonly SemaphoreSlim SendLock; + private readonly ConcurrentDictionary<int, FBMRequest> ActiveRequests; + private readonly ReusableStore<FBMRequest> RequestRental; + private readonly FBMRequest _controlFrame; + /// <summary> + /// The configuration for the current client + /// </summary> + public FBMClientConfig Config { get; } + /// <summary> + /// A handle that is reset when a connection has been successfully set, and is set + /// when the connection exists + /// </summary> + public ManualResetEvent ConnectionStatusHandle { get; } + /// <summary> + /// The <see cref="ClientWebSocket"/> to send/recieve message on + /// </summary> + public ManagedClientWebSocket ClientSocket { get; } + /// <summary> + /// Gets the shared control frame for the current instance. The request is reset when + /// this property is called. (Not thread safe) + /// </summary> + protected FBMRequest ControlFrame + { + get + { + _controlFrame.Reset(); + return _controlFrame; + } + } + + /// <summary> + /// Creates a new <see cref="FBMClient"/> in a closed state + /// </summary> + /// <param name="config">The client configuration</param> + public FBMClient(FBMClientConfig config) + { + RequestRental = ObjectRental.CreateReusable(ReuseableRequestConstructor); + SendLock = new(1); + ConnectionStatusHandle = new(true); + ActiveRequests = new(Environment.ProcessorCount, 100); + + Config = config; + //Init control frame + _controlFrame = new (Helpers.CONTROL_FRAME_MID, in config); + //Init the new client socket + ClientSocket = new(config.RecvBufferSize, config.RecvBufferSize, config.KeepAliveInterval, config.SubProtocol); + } + + private void Debug(string format, params string[] strings) + { + if(Config.DebugLog != null) + { + Config.DebugLog.Debug($"[DEBUG] FBM Client: {format}", strings); + } + } + private void Debug(string format, long value, long other) + { + if (Config.DebugLog != null) + { + Config.DebugLog.Debug($"[DEBUG] FBM Client: {format}", value, other); + } + } + + /// <summary> + /// Allocates and configures a new <see cref="FBMRequest"/> message object for use within the reusable store + /// </summary> + /// <returns>The configured <see cref="FBMRequest"/></returns> + protected virtual FBMRequest ReuseableRequestConstructor() => new(Config); + + /// <summary> + /// Asynchronously opens a websocket connection with the specifed remote server + /// </summary> + /// <param name="address">The address of the server to connect to</param> + /// <param name="cancellation">A cancellation token</param> + /// <returns></returns> + public async Task ConnectAsync(Uri address, CancellationToken cancellation = default) + { + //Uribuilder to send config parameters to the server + UriBuilder urib = new(address); + urib.Query += + $"{REQ_RECV_BUF_QUERY_ARG}={Config.RecvBufferSize}" + + $"&{REQ_HEAD_BUF_QUERY_ARG}={Config.MaxHeaderBufferSize}" + + $"&{REQ_MAX_MESS_QUERY_ARG}={Config.MaxMessageSize}"; + Debug("Connection string {con}", urib.Uri.ToString()); + //Connect to server + await ClientSocket.ConnectAsync(urib.Uri, cancellation); + //Reset wait handle before return + ConnectionStatusHandle.Reset(); + //Begin listeing for requets in a background task + _ = Task.Run(ProcessContinuousRecvAsync, cancellation); + } + + /// <summary> + /// Rents a new <see cref="FBMRequest"/> from the internal <see cref="ReusableStore{T}"/>. + /// Use <see cref="ReturnRequest(FBMRequest)"/> when request is no longer in use + /// </summary> + /// <returns>The configured (rented or new) <see cref="FBMRequest"/> ready for use</returns> + public FBMRequest RentRequest() => RequestRental.Rent(); + /// <summary> + /// Stores (or returns) the reusable request in cache for use with <see cref="RentRequest"/> + /// </summary> + /// <param name="request">The request to return to the store</param> + /// <exception cref="InvalidOperationException"></exception> + public void ReturnRequest(FBMRequest request) => RequestRental.Return(request); + + /// <summary> + /// Sends a <see cref="FBMRequest"/> to the connected server + /// </summary> + /// <param name="request">The request message to send to the server</param> + /// <param name="cancellationToken"></param> + /// <returns>When awaited, yields the server response</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="FBMInvalidRequestException"></exception> + public async Task<FBMResponse> SendAsync(FBMRequest request, CancellationToken cancellationToken = default) + { + Check(); + //Length of the request must contains at least 1 int and header byte + if (request.Length < 1 + sizeof(int)) + { + throw new FBMInvalidRequestException("Message is not initialized"); + } + //Store a null value in the request queue so the response can store a buffer + if (!ActiveRequests.TryAdd(request.MessageId, request)) + { + throw new ArgumentException("Message with the same ID is already being processed"); + } + try + { + Debug("Sending {bytes} with id {id}", request.RequestData.Length, request.MessageId); + + //Reset the wait handle + request.ResponseWaitEvent.Reset(); + + //Wait for send-lock + using (SemSlimReleaser releaser = await SendLock.GetReleaserAsync(cancellationToken)) + { + //Send the data to the server + await ClientSocket.SendAsync(request.RequestData, WebSocketMessageType.Binary, true, cancellationToken); + } + + //wait for the response to be set + await request.WaitForResponseAsync(cancellationToken); + + Debug("Received {size} bytes for message {id}", request.Response?.Length ?? 0, request.MessageId); + + return request.GetResponse(); + } + catch + { + //Remove the request since packet was never sent + ActiveRequests.Remove(request.MessageId, out _); + //Clear waiting flag + request.ResponseWaitEvent.Set(); + throw; + } + } + /// <summary> + /// Streams arbitrary binary data to the server with the initial request message + /// </summary> + /// <param name="request">The request message to send to the server</param> + /// <param name="payload">Data to stream to the server</param> + /// <param name="ct">The content type of the stream of data</param> + /// <param name="cancellationToken"></param> + /// <returns>When awaited, yields the server response</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public async Task StreamDataAsync(FBMRequest request, Stream payload, ContentType ct, CancellationToken cancellationToken = default) + { + Check(); + //Length of the request must contains at least 1 int and header byte + if(request.Length < 1 + sizeof(int)) + { + throw new FBMInvalidRequestException("Message is not initialized"); + } + //Store a null value in the request queue so the response can store a buffer + if (!ActiveRequests.TryAdd(request.MessageId, request)) + { + throw new ArgumentException("Message with the same ID is already being processed"); + } + try + { + Debug("Streaming {bytes} with id {id}", request.RequestData.Length, request.MessageId); + //Reset the wait handle + request.ResponseWaitEvent.Reset(); + //Write an empty body in the request + request.WriteBody(ReadOnlySpan<byte>.Empty, ct); + //Wait for send-lock + using (SemSlimReleaser releaser = await SendLock.GetReleaserAsync(cancellationToken)) + { + //Send the initial request packet + await ClientSocket.SendAsync(request.RequestData, WebSocketMessageType.Binary, false, cancellationToken); + //Calc buffer size + int bufSize = (int)Math.Clamp(payload.Length, Config.MessageBufferSize, Config.MaxMessageSize); + //Alloc a streaming buffer + using IMemoryOwner<byte> buffer = Config.BufferHeap.DirectAlloc<byte>(bufSize); + //Stream mesage body + do + { + //Read data + int read = await payload.ReadAsync(buffer.Memory, cancellationToken); + if (read == 0) + { + //No more data avialable + break; + } + //write message to socket, if the read data was smaller than the buffer, we can send the last packet + await ClientSocket.SendAsync(buffer.Memory[..read], WebSocketMessageType.Binary, read < bufSize, cancellationToken); + + } while (true); + } + //wait for the server to respond + await request.WaitForResponseAsync(cancellationToken); + + Debug("Response recieved {size} bytes for message {id}", request.Response?.Length ?? 0, request.MessageId); + } + catch + { + //Remove the request since packet was never sent or cancelled + ActiveRequests.Remove(request.MessageId, out _); + //Clear wait lock so the request state is reset + request.ResponseWaitEvent.Set(); + throw; + } + } + + /// <summary> + /// Begins listening for messages from the server on the internal socket (must be connected), + /// until the socket is closed, or canceled + /// </summary> + /// <returns></returns> + protected async Task ProcessContinuousRecvAsync() + { + Debug("Begining receive loop"); + //Alloc recv buffer + IMemoryOwner<byte> recvBuffer = Config.BufferHeap.DirectAlloc<byte>(Config.RecvBufferSize); + try + { + //Recv event loop + while (true) + { + //Listen for incoming packets with the intial data buffer + ValueWebSocketReceiveResult result = await ClientSocket.ReceiveAsync(recvBuffer.Memory, CancellationToken.None); + //If the message is a close message, its time to exit + if (result.MessageType == WebSocketMessageType.Close) + { + //Notify the event handler that the connection was closed + ConnectionClosed?.Invoke(this, EventArgs.Empty); + break; + } + if (result.Count <= 4) + { + Debug("Empty message recieved from server"); + continue; + } + //Alloc data buffer and write initial data + VnMemoryStream responseBuffer = new(Config.BufferHeap); + //Copy initial data + responseBuffer.Write(recvBuffer.Memory.Span[..result.Count]); + //Receive packets until the EOF is reached + while (!result.EndOfMessage) + { + //recive more data + result = await ClientSocket.ReceiveAsync(recvBuffer.Memory, CancellationToken.None); + //Make sure the buffer is not too large + if ((responseBuffer.Length + result.Count) > Config.MaxMessageSize) + { + //Dispose the buffer before exiting + responseBuffer.Dispose(); + Debug("Recieved a message that was too large, skipped"); + goto Skip; + } + //Copy continuous data + responseBuffer.Write(recvBuffer.Memory.Span[..result.Count]); + } + //Reset the buffer stream position + _ = responseBuffer.Seek(0, SeekOrigin.Begin); + ProcessResponse(responseBuffer); + //Goto skip statment to cleanup resources + Skip:; + } + } + catch (OperationCanceledException) + { + //Normal closeure, do nothing + } + catch (Exception ex) + { + //Error event args + FMBClientErrorEventArgs wsEventArgs = new() + { + Cause = ex, + ErrorClient = this + }; + //Invoke error handler + ConnectionClosedOnError?.Invoke(this, wsEventArgs); + } + finally + { + //Dispose the recv buffer + recvBuffer.Dispose(); + //Set all pending events + foreach (FBMRequest request in ActiveRequests.Values) + { + request.ResponseWaitEvent.Set(); + } + //Clear dict + ActiveRequests.Clear(); + //Cleanup the socket when exiting + ClientSocket.Cleanup(); + //Set status handle as unset + ConnectionStatusHandle.Set(); + //Invoke connection closed + ConnectionClosed?.Invoke(this, EventArgs.Empty); + } + Debug("Receive loop exited"); + } + + /// <summary> + /// Syncrhonously processes a buffered response packet + /// </summary> + /// <param name="responseMessage">The buffered response body recieved from the server</param> + /// <remarks>This method blocks the listening task. So operations should be quick</remarks> + protected virtual void ProcessResponse(VnMemoryStream responseMessage) + { + //read first response line + ReadOnlySpan<byte> line = Helpers.ReadLine(responseMessage); + //get the id of the message + int messageId = Helpers.GetMessageId(line); + //Finalze control frame + if(messageId == Helpers.CONTROL_FRAME_MID) + { + Debug("Control frame received"); + ProcessControlFrame(responseMessage); + return; + } + else if (messageId < 0) + { + //Cannot process request + responseMessage.Dispose(); + Debug("Invalid messageid"); + return; + } + //Search for the request that has the same id + if (ActiveRequests.TryRemove(messageId, out FBMRequest? request)) + { + //Set the new response message + request.SetResponse(responseMessage); + } + else + { + Debug("Message {id} was not found in the waiting message queue", messageId, 0); + + //Cleanup no request was waiting + responseMessage.Dispose(); + } + } + /// <summary> + /// Processes a control frame response from the server + /// </summary> + /// <param name="vms">The raw response packet from the server</param> + private void ProcessControlFrame(VnMemoryStream vms) + { + vms.Dispose(); + } + /// <summary> + /// Processes a control frame response from the server + /// </summary> + /// <param name="response">The parsed response-packet</param> + protected virtual void ProcessControlFrame(in FBMResponse response) + { + + } + + /// <summary> + /// Closes the underlying <see cref="WebSocket"/> and cancels all pending operations + /// </summary> + /// <param name="cancellationToken"></param> + /// <returns></returns> + /// <exception cref="ObjectDisposedException"></exception> + public async Task DisconnectAsync(CancellationToken cancellationToken = default) + { + Check(); + //Close the connection + await ClientSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "Closing", cancellationToken); + } + ///<inheritdoc/> + protected override void Free() + { + //Dispose socket + ClientSocket.Dispose(); + //Dispose client buffer + RequestRental.Dispose(); + SendLock.Dispose(); + ConnectionStatusHandle.Dispose(); + } + ///<inheritdoc/> + public void CacheClear() => RequestRental.CacheClear(); + ///<inheritdoc/> + public void CacheHardClear() => RequestRental.CacheHardClear(); + } +} diff --git a/Net.Messaging.FBM/src/Client/FBMClientConfig.cs b/Net.Messaging.FBM/src/Client/FBMClientConfig.cs new file mode 100644 index 0000000..229eb76 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/FBMClientConfig.cs @@ -0,0 +1,81 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMClientConfig.cs +* +* FBMClientConfig.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; + +using VNLib.Utils.Memory; +using VNLib.Utils.Logging; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// A structure that defines readonly constants for the <see cref="FBMClient"/> to use + /// </summary> + public readonly struct FBMClientConfig + { + /// <summary> + /// The size (in bytes) of the internal buffer to use when receiving messages from the server + /// </summary> + public readonly int RecvBufferSize { get; init; } + /// <summary> + /// The size (in bytes) of the <see cref="FBMRequest"/> internal buffer size, when requests are rented from the client + /// </summary> + /// <remarks> + /// This is the entire size of the request buffer including headers and payload data, unless + /// data is streamed to the server + /// </remarks> + public readonly int MessageBufferSize { get; init; } + /// <summary> + /// The size (in chars) of the client/server message header buffer + /// </summary> + public readonly int MaxHeaderBufferSize { get; init; } + /// <summary> + /// The maximum size (in bytes) of messages sent or recieved from the server + /// </summary> + public readonly int MaxMessageSize { get; init; } + /// <summary> + /// The heap to allocate interal (and message) buffers from + /// </summary> + public readonly IUnmangedHeap BufferHeap { get; init; } + /// <summary> + /// The websocket keepalive interval to use (leaving this property default disables keepalives) + /// </summary> + public readonly TimeSpan KeepAliveInterval { get; init; } + /// <summary> + /// The websocket sub-protocol to use + /// </summary> + public readonly string? SubProtocol { get; init; } + /// <summary> + /// The encoding instance used to encode header values + /// </summary> + public readonly Encoding HeaderEncoding { get; init; } + + /// <summary> + /// An optional log provider to write debug logs to. If this propery is not null, + /// debugging information will be logged with the debug log-level + /// </summary> + public readonly ILogProvider? DebugLog { get; init; } + } +} diff --git a/Net.Messaging.FBM/src/Client/FBMClientWorkerBase.cs b/Net.Messaging.FBM/src/Client/FBMClientWorkerBase.cs new file mode 100644 index 0000000..b4056dc --- /dev/null +++ b/Net.Messaging.FBM/src/Client/FBMClientWorkerBase.cs @@ -0,0 +1,125 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMClientWorkerBase.cs +* +* FBMClientWorkerBase.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// A base class for objects that implement <see cref="FBMClient"/> + /// operations + /// </summary> + public abstract class FBMClientWorkerBase : VnDisposeable, IStatefulConnection + { + /// <summary> + /// Allows configuration of websocket configuration options + /// </summary> + public ManagedClientWebSocket SocketOptions => Client.ClientSocket; + +#nullable disable + /// <summary> + /// The <see cref="FBMClient"/> to sent requests from + /// </summary> + public FBMClient Client { get; private set; } + + /// <summary> + /// Raised when the client has connected successfully + /// </summary> + public event Action<FBMClient, FBMClientWorkerBase> Connected; +#nullable enable + + ///<inheritdoc/> + public event EventHandler ConnectionClosed + { + add => Client.ConnectionClosed += value; + remove => Client.ConnectionClosed -= value; + } + + /// <summary> + /// Creates and initializes a the internal <see cref="FBMClient"/> + /// </summary> + /// <param name="config">The client config</param> + protected void InitClient(in FBMClientConfig config) + { + Client = new(config); + Client.ConnectionClosedOnError += Client_ConnectionClosedOnError; + Client.ConnectionClosed += Client_ConnectionClosed; + } + + private void Client_ConnectionClosed(object? sender, EventArgs e) => OnDisconnected(); + private void Client_ConnectionClosedOnError(object? sender, FMBClientErrorEventArgs e) => OnError(e); + + /// <summary> + /// Asynchronously connects to a remote server by the specified uri + /// </summary> + /// <param name="serverUri">The remote uri of a server to connect to</param> + /// <param name="cancellationToken">A token to cancel the connect operation</param> + /// <returns>A task that compeltes when the client has connected to the remote server</returns> + public virtual async Task ConnectAsync(Uri serverUri, CancellationToken cancellationToken = default) + { + //Connect to server + await Client.ConnectAsync(serverUri, cancellationToken).ConfigureAwait(true); + //Invoke child on-connected event + OnConnected(); + Connected?.Invoke(Client, this); + } + + /// <summary> + /// Asynchronously disonnects a client only if the client is currently connected, + /// returns otherwise + /// </summary> + /// <param name="cancellationToken"></param> + /// <returns>A task that compeltes when the client has disconnected</returns> + public virtual Task DisconnectAsync(CancellationToken cancellationToken = default) + { + return Client.DisconnectAsync(cancellationToken); + } + + /// <summary> + /// Invoked when a client has successfully connected to the remote server + /// </summary> + protected abstract void OnConnected(); + /// <summary> + /// Invoked when the client has disconnected cleanly + /// </summary> + protected abstract void OnDisconnected(); + /// <summary> + /// Invoked when the connected client is closed because of a connection error + /// </summary> + /// <param name="e">A <see cref="EventArgs"/> that contains the client error data</param> + protected abstract void OnError(FMBClientErrorEventArgs e); + + ///<inheritdoc/> + protected override void Free() + { + Client.ConnectionClosedOnError -= Client_ConnectionClosedOnError; + Client.ConnectionClosed -= Client_ConnectionClosed; + Client.Dispose(); + } + } +} diff --git a/Net.Messaging.FBM/src/Client/FBMRequest.cs b/Net.Messaging.FBM/src/Client/FBMRequest.cs new file mode 100644 index 0000000..9d8af42 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/FBMRequest.cs @@ -0,0 +1,302 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMRequest.cs +* +* FBMRequest.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; +using System.Collections.Generic; + +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; +using VNLib.Utils.Memory.Caching; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// A reusable Fixed Buffer Message request container. This class is not thread-safe + /// </summary> + public sealed class FBMRequest : VnDisposeable, IReusable, IFBMMessage, IStringSerializeable + { + private class BufferWriter : IBufferWriter<byte> + { + private readonly FBMRequest _request; + + public BufferWriter(FBMRequest request) + { + _request = request; + } + + public void Advance(int count) + { + _request.Position += count; + } + + public Memory<byte> GetMemory(int sizeHint = 0) + { + return sizeHint > 0 ? _request.RemainingBuffer[0..sizeHint] : _request.RemainingBuffer; + } + + public Span<byte> GetSpan(int sizeHint = 0) + { + return sizeHint > 0 ? _request.RemainingBuffer.Span[0..sizeHint] : _request.RemainingBuffer.Span; + } + } + + private readonly IMemoryOwner<byte> HeapBuffer; + + + private readonly BufferWriter _writer; + private int Position; + + private readonly Encoding HeaderEncoding; + private readonly int ResponseHeaderBufferSize; + private readonly List<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>> ResponseHeaderList = new(); + private char[]? ResponseHeaderBuffer; + + /// <summary> + /// The size (in bytes) of the request message + /// </summary> + public int Length => Position; + private Memory<byte> RemainingBuffer => HeapBuffer.Memory[Position..]; + + /// <summary> + /// The id of the current request message + /// </summary> + public int MessageId { get; } + /// <summary> + /// The request message packet + /// </summary> + public ReadOnlyMemory<byte> RequestData => HeapBuffer.Memory[..Position]; + /// <summary> + /// An <see cref="ManualResetEvent"/> to signal request/response + /// event completion + /// </summary> + internal ManualResetEvent ResponseWaitEvent { get; } + + internal VnMemoryStream? Response { get; private set; } + /// <summary> + /// Initializes a new <see cref="FBMRequest"/> with the sepcified message buffer size, + /// and a random messageid + /// </summary> + /// <param name="config">The fbm client config storing required config variables</param> + public FBMRequest(in FBMClientConfig config) : this(Helpers.RandomMessageId, in config) + { } + /// <summary> + /// Initializes a new <see cref="FBMRequest"/> with the sepcified message buffer size and a custom MessageId + /// </summary> + /// <param name="messageId">The custom message id</param> + /// <param name="config">The fbm client config storing required config variables</param> + public FBMRequest(int messageId, in FBMClientConfig config) + { + //Setup response wait handle but make sure the contuation runs async + ResponseWaitEvent = new(true); + + //Alloc the buffer as a memory owner so a memory buffer can be used + HeapBuffer = config.BufferHeap.DirectAlloc<byte>(config.MessageBufferSize); + + MessageId = messageId; + + HeaderEncoding = config.HeaderEncoding; + ResponseHeaderBufferSize = config.MaxHeaderBufferSize; + + WriteMessageId(); + _writer = new(this); + } + + /// <summary> + /// Resets the internal buffer and writes the message-id header to the begining + /// of the buffer + /// </summary> + private void WriteMessageId() + { + //Get writer over buffer + ForwardOnlyWriter<byte> buffer = new(HeapBuffer.Memory.Span); + //write messageid header to the buffer + buffer.Append((byte)HeaderCommand.MessageId); + buffer.Append(MessageId); + buffer.WriteTermination(); + //Store intial position + Position = buffer.Written; + } + + ///<inheritdoc/> + public void WriteHeader(HeaderCommand header, ReadOnlySpan<char> value) => WriteHeader((byte)header, value); + ///<inheritdoc/> + public void WriteHeader(byte header, ReadOnlySpan<char> value) + { + ForwardOnlyWriter<byte> buffer = new(RemainingBuffer.Span); + buffer.WriteHeader(header, value, Helpers.DefaultEncoding); + //Update position + Position += buffer.Written; + } + ///<inheritdoc/> + public void WriteBody(ReadOnlySpan<byte> body, ContentType contentType = ContentType.Binary) + { + //Write content type header + WriteHeader(HeaderCommand.ContentType, HttpHelpers.GetContentTypeString(contentType)); + //Get writer over buffer + ForwardOnlyWriter<byte> buffer = new(RemainingBuffer.Span); + //Now safe to write body + buffer.WriteBody(body); + //Update position + Position += buffer.Written; + } + /// <summary> + /// Returns buffer writer for writing the body data to the internal message buffer + /// </summary> + /// <returns>A <see cref="BufferWriter"/> to write message body to</returns> + public IBufferWriter<byte> GetBodyWriter() + { + //Write body termination + Helpers.Termination.CopyTo(RemainingBuffer); + Position += Helpers.Termination.Length; + //Return buffer writer + return _writer; + } + + /// <summary> + /// Resets the internal buffer and allows for writing a new message with + /// the same message-id + /// </summary> + public void Reset() + { + //Re-writing the message-id will reset the buffer + WriteMessageId(); + } + + internal void SetResponse(VnMemoryStream? vms) + { + Response = vms; + ResponseWaitEvent.Set(); + } + + internal Task WaitForResponseAsync(CancellationToken token) + { + return ResponseWaitEvent.WaitAsync().WaitAsync(token); + } + + ///<inheritdoc/> + protected override void Free() + { + HeapBuffer.Dispose(); + ResponseWaitEvent.Dispose(); + OnResponseDisposed(); + } + void IReusable.Prepare() => Reset(); + bool IReusable.Release() + { + //Clear old response data if error occured + Response?.Dispose(); + Response = null; + + return true; + } + + /// <summary> + /// Gets the response of the sent message + /// </summary> + /// <returns>The response message for the current request</returns> + internal FBMResponse GetResponse() + { + if (Response != null) + { + /* + * NOTICE + * + * The FBM Client will position the response stream to the start + * of the header section (missing the message-id header) + * + * The message id belongs to this request so it cannot be mismatched + * + * The headers are read into a list of key-value pairs and the stream + * is positioned to the start of the message body + */ + + + //Alloc rseponse buffer + ResponseHeaderBuffer ??= ArrayPool<char>.Shared.Rent(ResponseHeaderBufferSize); + + //Parse message headers + HeaderParseError statusFlags = Helpers.ParseHeaders(Response, ResponseHeaderBuffer, ResponseHeaderList, HeaderEncoding); + + //return response structure + return new(Response, statusFlags, ResponseHeaderList, OnResponseDisposed); + } + else + { + return new(); + } + } + + //Called when a response message is disposed to cleanup resources held by the response + private void OnResponseDisposed() + { + //Clear response header list + ResponseHeaderList.Clear(); + + //Clear old response + Response?.Dispose(); + Response = null; + + if (ResponseHeaderBuffer != null) + { + //Free response buffer + ArrayPool<char>.Shared.Return(ResponseHeaderBuffer!); + ResponseHeaderBuffer = null; + } + } + + ///<inheritdoc/> + public string Compile() + { + int charSize = Helpers.DefaultEncoding.GetCharCount(RequestData.Span); + using UnsafeMemoryHandle<char> buffer = Memory.UnsafeAlloc<char>(charSize + 128); + ERRNO count = Compile(buffer.Span); + return buffer.AsSpan(0, count).ToString(); + } + ///<inheritdoc/> + public void Compile(ref ForwardOnlyWriter<char> writer) + { + writer.Append("Message ID:"); + writer.Append(MessageId); + writer.Append(Environment.NewLine); + Helpers.DefaultEncoding.GetChars(RequestData.Span, ref writer); + } + ///<inheritdoc/> + public ERRNO Compile(in Span<char> buffer) + { + ForwardOnlyWriter<char> writer = new(buffer); + Compile(ref writer); + return writer.Written; + } + ///<inheritdoc/> + public override string ToString() => Compile(); + + } +} diff --git a/Net.Messaging.FBM/src/Client/FBMResponse.cs b/Net.Messaging.FBM/src/Client/FBMResponse.cs new file mode 100644 index 0000000..da36956 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/FBMResponse.cs @@ -0,0 +1,106 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMResponse.cs +* +* FBMResponse.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; + +using VNLib.Utils.IO; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// A Fixed Buffer Message client response linked to the request that generated it. + /// Once the request is disposed or returned this message state is invalid + /// </summary> + public readonly struct FBMResponse : IDisposable, IEquatable<FBMResponse> + { + private readonly Action? _onDispose; + + /// <summary> + /// True when a response body was recieved and properly parsed + /// </summary> + public readonly bool IsSet { get; } + /// <summary> + /// The raw response message packet + /// </summary> + public readonly VnMemoryStream? MessagePacket { get; } + /// <summary> + /// A collection of response message headers + /// </summary> + public readonly IReadOnlyList<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>> Headers { get; } + /// <summary> + /// Status flags of the message parse operation + /// </summary> + public readonly HeaderParseError StatusFlags { get; } + /// <summary> + /// The body segment of the response message + /// </summary> + public readonly ReadOnlySpan<byte> ResponseBody => IsSet ? Helpers.GetRemainingData(MessagePacket!) : ReadOnlySpan<byte>.Empty; + + /// <summary> + /// Initailzies a response message structure and parses response + /// packet structure + /// </summary> + /// <param name="vms">The message buffer (message packet)</param> + /// <param name="status">The size of the buffer to alloc for header value storage</param> + /// <param name="headerList">The collection of headerse</param> + /// <param name="onDispose">A method that will be invoked when the message response body is disposed</param> + public FBMResponse(VnMemoryStream? vms, HeaderParseError status, IReadOnlyList<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>> headerList, Action onDispose) + { + MessagePacket = vms; + StatusFlags = status; + Headers = headerList; + IsSet = true; + _onDispose = onDispose; + } + + /// <summary> + /// Creates an unset response structure + /// </summary> + public FBMResponse() + { + MessagePacket = null; + StatusFlags = HeaderParseError.InvalidHeaderRead; + Headers = Array.Empty<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>>(); + IsSet = false; + _onDispose = null; + } + + /// <summary> + /// Releases any resources associated with the response message + /// </summary> + public void Dispose() => _onDispose?.Invoke(); + ///<inheritdoc/> + public override bool Equals(object? obj) => obj is FBMResponse response && Equals(response); + ///<inheritdoc/> + public override int GetHashCode() => IsSet ? MessagePacket!.GetHashCode() : 0; + ///<inheritdoc/> + public static bool operator ==(FBMResponse left, FBMResponse right) => left.Equals(right); + ///<inheritdoc/> + public static bool operator !=(FBMResponse left, FBMResponse right) => !(left == right); + ///<inheritdoc/> + public bool Equals(FBMResponse other) => IsSet && other.IsSet && MessagePacket == other.MessagePacket; + + } +} diff --git a/Net.Messaging.FBM/src/Client/FMBClientErrorEventArgs.cs b/Net.Messaging.FBM/src/Client/FMBClientErrorEventArgs.cs new file mode 100644 index 0000000..96e9414 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/FMBClientErrorEventArgs.cs @@ -0,0 +1,46 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FMBClientErrorEventArgs.cs +* +* FMBClientErrorEventArgs.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +#nullable disable + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// <see cref="EventArgs"/> that is raised when an error occurs + /// in the background listener loop + /// </summary> + public class FMBClientErrorEventArgs : EventArgs + { + /// <summary> + /// The client that the exception was raised from + /// </summary> + public FBMClient ErrorClient { get; init; } + /// <summary> + /// The exception that was raised + /// </summary> + public Exception Cause { get; init; } + } +} diff --git a/Net.Messaging.FBM/src/Client/HeaderCommand.cs b/Net.Messaging.FBM/src/Client/HeaderCommand.cs new file mode 100644 index 0000000..5a57d85 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/HeaderCommand.cs @@ -0,0 +1,57 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: HeaderCommand.cs +* +* HeaderCommand.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// A Fixed Buffer Message header command value + /// </summary> + public enum HeaderCommand : byte + { + /// <summary> + /// Default, do not use + /// </summary> + NOT_USED, + /// <summary> + /// Specifies the header for a message-id + /// </summary> + MessageId, + /// <summary> + /// Specifies a resource location + /// </summary> + Location, + /// <summary> + /// Specifies a standard MIME content type header + /// </summary> + ContentType, + /// <summary> + /// Specifies an action on a request + /// </summary> + Action, + /// <summary> + /// Specifies a status header + /// </summary> + Status + } +} diff --git a/Net.Messaging.FBM/src/Client/HeaderParseStatus.cs b/Net.Messaging.FBM/src/Client/HeaderParseStatus.cs new file mode 100644 index 0000000..d38df26 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/HeaderParseStatus.cs @@ -0,0 +1,40 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: HeaderParseStatus.cs +* +* HeaderParseStatus.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// Specifies the results of a response parsing operation + /// </summary> + [Flags] + public enum HeaderParseError + { + None = 0, + InvalidId = 1, + HeaderOutOfMem = 2, + InvalidHeaderRead = 4 + } +} diff --git a/Net.Messaging.FBM/src/Client/Helpers.cs b/Net.Messaging.FBM/src/Client/Helpers.cs new file mode 100644 index 0000000..8f895fa --- /dev/null +++ b/Net.Messaging.FBM/src/Client/Helpers.cs @@ -0,0 +1,272 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: Helpers.cs +* +* Helpers.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Collections.Generic; +using System.Security.Cryptography; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// Contains FBM library helper methods + /// </summary> + public static class Helpers + { + /// <summary> + /// The message-id of a connection control frame / out of band message + /// </summary> + public const int CONTROL_FRAME_MID = -500; + + public static readonly Encoding DefaultEncoding = Encoding.UTF8; + public static readonly ReadOnlyMemory<byte> Termination = new byte[] { 0xFF, 0xF1 }; + + /// <summary> + /// Parses the header line for a message-id + /// </summary> + /// <param name="line">A sequence of bytes that make up a header line</param> + /// <returns>The message-id if parsed, -1 if message-id is not valid</returns> + public static int GetMessageId(ReadOnlySpan<byte> line) + { + //Make sure the message line is large enough to contain a message-id + if (line.Length < 1 + sizeof(int)) + { + return -1; + } + //The first byte should be the header id + HeaderCommand headerId = (HeaderCommand)line[0]; + //Make sure the headerid is set + if (headerId != HeaderCommand.MessageId) + { + return -2; + } + //Get the messageid after the header byte + ReadOnlySpan<byte> messageIdSegment = line.Slice(1, sizeof(int)); + //get the messageid from the messageid segment + return BitConverter.ToInt32(messageIdSegment); + } + + /// <summary> + /// Alloctes a random integer to use as a message id + /// </summary> + public static int RandomMessageId => RandomNumberGenerator.GetInt32(1, int.MaxValue); + /// <summary> + /// Gets the remaining data after the current position of the stream. + /// </summary> + /// <param name="response">The stream to segment</param> + /// <returns>The remaining data segment</returns> + public static ReadOnlySpan<byte> GetRemainingData(VnMemoryStream response) + { + return response.AsSpan()[(int)response.Position..]; + } + + /// <summary> + /// Reads the next available line from the response message + /// </summary> + /// <param name="response"></param> + /// <returns>The read line</returns> + public static ReadOnlySpan<byte> ReadLine(VnMemoryStream response) + { + //Get the span from the current stream position to end of the stream + ReadOnlySpan<byte> line = GetRemainingData(response); + //Search for next line termination + int index = line.IndexOf(Termination.Span); + if (index == -1) + { + return ReadOnlySpan<byte>.Empty; + } + //Update stream position to end of termination + response.Seek(index + Termination.Length, SeekOrigin.Current); + //slice up line and exclude the termination + return line[..index]; + } + /// <summary> + /// Parses headers from the request stream, stores headers from the buffer into the + /// header collection + /// </summary> + /// <param name="vms">The FBM packet buffer</param> + /// <param name="buffer">The header character buffer to write headers to</param> + /// <param name="headers">The collection to store headers in</param> + /// <param name="encoding">The encoding type used to deocde header values</param> + /// <returns>The results of the parse operation</returns> + public static HeaderParseError ParseHeaders(VnMemoryStream vms, char[] buffer, ICollection<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>> headers, Encoding encoding) + { + HeaderParseError status = HeaderParseError.None; + //sliding window + Memory<char> currentWindow = buffer; + //Accumulate headers + while (true) + { + //Read the next line from the current stream + ReadOnlySpan<byte> line = ReadLine(vms); + if (line.IsEmpty) + { + //Done reading headers + break; + } + HeaderCommand cmd = GetHeaderCommand(line); + //Get header value + ERRNO charsRead = GetHeaderValue(line, currentWindow.Span, encoding); + if (charsRead < 0) + { + //Out of buffer space + status |= HeaderParseError.HeaderOutOfMem; + break; + } + else if (!charsRead) + { + //Invalid header + status |= HeaderParseError.InvalidHeaderRead; + } + else + { + //Store header as a read-only sequence + headers.Add(new(cmd, currentWindow[..(int)charsRead])); + //Shift buffer window + currentWindow = currentWindow[(int)charsRead..]; + } + } + return status; + } + + /// <summary> + /// Gets a <see cref="HeaderCommand"/> enum from the first byte of the message + /// </summary> + /// <param name="line"></param> + /// <returns>The <see cref="HeaderCommand"/> enum value from hte first byte of the message</returns> + public static HeaderCommand GetHeaderCommand(ReadOnlySpan<byte> line) + { + return (HeaderCommand)line[0]; + } + /// <summary> + /// Gets the value of the header following the colon bytes in the specifed + /// data message data line + /// </summary> + /// <param name="line">The message header line to get the value of</param> + /// <param name="output">The output character buffer to write characters to</param> + /// <param name="encoding">The encoding to decode the specified data with</param> + /// <returns>The number of characters encoded</returns> + public static ERRNO GetHeaderValue(ReadOnlySpan<byte> line, Span<char> output, Encoding encoding) + { + //Get the data following the header byte + ReadOnlySpan<byte> value = line[1..]; + //Calculate the character account + int charCount = encoding.GetCharCount(value); + //Determine if the output buffer is large enough + if (charCount > output.Length) + { + return -1; + } + //Decode the characters and return the char count + _ = encoding.GetChars(value, output); + return charCount; + } + + /// <summary> + /// Appends an arbitrary header to the current request buffer + /// </summary> + /// <param name="buffer"></param> + /// <param name="header">The <see cref="HeaderCommand"/> of the header</param> + /// <param name="value">The value of the header</param> + /// <param name="encoding">Encoding to use when writing character message</param> + /// <exception cref="OutOfMemoryException"></exception> + public static void WriteHeader(ref this ForwardOnlyWriter<byte> buffer, byte header, ReadOnlySpan<char> value, Encoding encoding) + { + //get char count + int byteCount = encoding.GetByteCount(value); + //make sure there is enough room in the buffer + if (buffer.RemainingSize < byteCount) + { + throw new OutOfMemoryException(); + } + //Write header command enum value + buffer.Append(header); + //Convert the characters to binary and write to the buffer + encoding.GetBytes(value, ref buffer); + //Write termination (0) + buffer.WriteTermination(); + } + + /// <summary> + /// Ends the header section of the request and appends the message body to + /// the end of the request + /// </summary> + /// <param name="buffer"></param> + /// <param name="body">The message body to send with request</param> + /// <exception cref="OutOfMemoryException"></exception> + public static void WriteBody(ref this ForwardOnlyWriter<byte> buffer, ReadOnlySpan<byte> body) + { + //start with termination + buffer.WriteTermination(); + //Write the body + buffer.Append(body); + } + /// <summary> + /// Writes a line termination to the message buffer + /// </summary> + /// <param name="buffer"></param> + public static void WriteTermination(ref this ForwardOnlyWriter<byte> buffer) + { + //write termination + buffer.Append(Termination.Span); + } + + /// <summary> + /// Writes a line termination to the message buffer + /// </summary> + /// <param name="buffer"></param> + public static void WriteTermination(this IDataAccumulator<byte> buffer) + { + //write termination + buffer.Append(Termination.Span); + } + + /// <summary> + /// Appends an arbitrary header to the current request buffer + /// </summary> + /// <param name="buffer"></param> + /// <param name="header">The <see cref="HeaderCommand"/> of the header</param> + /// <param name="value">The value of the header</param> + /// <param name="encoding">Encoding to use when writing character message</param> + /// <exception cref="ArgumentException"></exception> + public static void WriteHeader(this IDataAccumulator<byte> buffer, byte header, ReadOnlySpan<char> value, Encoding encoding) + { + //Write header command enum value + buffer.Append(header); + //Convert the characters to binary and write to the buffer + int written = encoding.GetBytes(value, buffer.Remaining); + //Advance the buffer + buffer.Advance(written); + //Write termination (0) + buffer.WriteTermination(); + } + } +} diff --git a/Net.Messaging.FBM/src/Client/IFBMMessage.cs b/Net.Messaging.FBM/src/Client/IFBMMessage.cs new file mode 100644 index 0000000..18f19ec --- /dev/null +++ b/Net.Messaging.FBM/src/Client/IFBMMessage.cs @@ -0,0 +1,62 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: IFBMMessage.cs +* +* IFBMMessage.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Net.Http; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// Represents basic Fixed Buffer Message protocol operations + /// </summary> + public interface IFBMMessage + { + /// <summary> + /// The unique id of the message (nonzero) + /// </summary> + int MessageId { get; } + /// <summary> + /// Writes a data body to the message of the specified content type + /// </summary> + /// <param name="body">The body of the message to copy</param> + /// <param name="contentType">The content type of the message body</param> + /// <exception cref="OutOfMemoryException"></exception> + void WriteBody(ReadOnlySpan<byte> body, ContentType contentType = ContentType.Binary); + /// <summary> + /// Appends an arbitrary header to the current request buffer + /// </summary> + /// <param name="header">The header id</param> + /// <param name="value">The value of the header</param> + /// <exception cref="OutOfMemoryException"></exception> + void WriteHeader(byte header, ReadOnlySpan<char> value); + /// <summary> + /// Appends an arbitrary header to the current request buffer + /// </summary> + /// <param name="header">The <see cref="HeaderCommand"/> of the header</param> + /// <param name="value">The value of the header</param> + /// <exception cref="OutOfMemoryException"></exception> + void WriteHeader(HeaderCommand header, ReadOnlySpan<char> value); + } +}
\ No newline at end of file diff --git a/Net.Messaging.FBM/src/Client/IStatefulConnection.cs b/Net.Messaging.FBM/src/Client/IStatefulConnection.cs new file mode 100644 index 0000000..3b9dd3b --- /dev/null +++ b/Net.Messaging.FBM/src/Client/IStatefulConnection.cs @@ -0,0 +1,54 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: IStatefulConnection.cs +* +* IStatefulConnection.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Messaging.FBM.Client +{ + /// <summary> + /// An abstraction for a stateful connection client that reports its status + /// </summary> + public interface IStatefulConnection + { + /// <summary> + /// An event that is raised when the connection state has transition from connected to disconnected + /// </summary> + event EventHandler ConnectionClosed; + /// <summary> + /// Connects the client to the remote resource + /// </summary> + /// <param name="serverUri">The resource location to connect to</param> + /// <param name="cancellationToken">A token to cancel the connect opreation</param> + /// <returns>A task that compeltes when the connection has succedded</returns> + Task ConnectAsync(Uri serverUri, CancellationToken cancellationToken = default); + /// <summary> + /// Gracefully disconnects the client from the remote resource + /// </summary> + /// <param name="cancellationToken">A token to cancel the disconnect operation</param> + /// <returns>A task that completes when the client has been disconnected</returns> + Task DisconnectAsync(CancellationToken cancellationToken = default); + } +} diff --git a/Net.Messaging.FBM/src/Client/ManagedClientWebSocket.cs b/Net.Messaging.FBM/src/Client/ManagedClientWebSocket.cs new file mode 100644 index 0000000..acac369 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/ManagedClientWebSocket.cs @@ -0,0 +1,201 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: ManagedClientWebSocket.cs +* +* ManagedClientWebSocket.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Threading; +using System.Net.Security; +using System.Net.WebSockets; +using System.Threading.Tasks; +using System.Security.Cryptography.X509Certificates; + +using VNLib.Utils.Memory; + +#nullable enable + +namespace VNLib.Net.Messaging.FBM.Client +{ + + /// <summary> + /// A wrapper container to manage client websockets + /// </summary> + public class ManagedClientWebSocket : WebSocket + { + private readonly int TxBufferSize; + private readonly int RxBufferSize; + private readonly TimeSpan KeepAliveInterval; + private readonly VnTempBuffer<byte> _dataBuffer; + private readonly string? _subProtocol; + + /// <summary> + /// A collection of headers to add to the client + /// </summary> + public WebHeaderCollection Headers { get; } + public X509CertificateCollection Certificates { get; } + public IWebProxy? Proxy { get; set; } + public CookieContainer? Cookies { get; set; } + public RemoteCertificateValidationCallback? RemoteCertificateValidationCallback { get; set; } + + + private ClientWebSocket? _socket; + + /// <summary> + /// Initiaizes a new <see cref="ManagedClientWebSocket"/> that accepts an optional sub-protocol for connections + /// </summary> + /// <param name="txBufferSize">The size (in bytes) of the send buffer size</param> + /// <param name="rxBufferSize">The size (in bytes) of the receive buffer size to use</param> + /// <param name="keepAlive">The WS keepalive interval</param> + /// <param name="subProtocol">The optional sub-protocol to use</param> + public ManagedClientWebSocket(int txBufferSize, int rxBufferSize, TimeSpan keepAlive, string? subProtocol) + { + //Init header collection + Headers = new(); + Certificates = new(); + //Alloc buffer + _dataBuffer = new(rxBufferSize); + TxBufferSize = txBufferSize; + RxBufferSize = rxBufferSize; + KeepAliveInterval = keepAlive; + _subProtocol = subProtocol; + } + + /// <summary> + /// Asyncrhonously prepares a new client web-socket and connects to the remote endpoint + /// </summary> + /// <param name="serverUri">The endpoint to connect to</param> + /// <param name="token">A token to cancel the connect operation</param> + /// <returns>A task that compeltes when the client has connected</returns> + public async Task ConnectAsync(Uri serverUri, CancellationToken token) + { + //Init new socket + _socket = new(); + try + { + //Set buffer + _socket.Options.SetBuffer(RxBufferSize, TxBufferSize, _dataBuffer); + //Set remaining stored options + _socket.Options.ClientCertificates = Certificates; + _socket.Options.KeepAliveInterval = KeepAliveInterval; + _socket.Options.Cookies = Cookies; + _socket.Options.Proxy = Proxy; + _socket.Options.RemoteCertificateValidationCallback = RemoteCertificateValidationCallback; + //Specify sub-protocol + if (!string.IsNullOrEmpty(_subProtocol)) + { + _socket.Options.AddSubProtocol(_subProtocol); + } + //Set headers + for (int i = 0; i < Headers.Count; i++) + { + string name = Headers.GetKey(i); + string? value = Headers.Get(i); + //Set header + _socket.Options.SetRequestHeader(name, value); + } + //Connect to server + await _socket.ConnectAsync(serverUri, token); + } + catch + { + //Cleanup the socket + Cleanup(); + throw; + } + } + + /// <summary> + /// Cleans up internal resources to prepare for another connection + /// </summary> + public void Cleanup() + { + //Dispose old socket if set + _socket?.Dispose(); + _socket = null; + } + ///<inheritdoc/> + public override WebSocketCloseStatus? CloseStatus => _socket?.CloseStatus; + ///<inheritdoc/> + public override string CloseStatusDescription => _socket?.CloseStatusDescription ?? string.Empty; + ///<inheritdoc/> + public override WebSocketState State => _socket?.State ?? WebSocketState.Closed; + ///<inheritdoc/> + public override string SubProtocol => _subProtocol ?? string.Empty; + + + ///<inheritdoc/> + public override void Abort() + { + _socket?.Abort(); + } + ///<inheritdoc/> + public override Task CloseAsync(WebSocketCloseStatus closeStatus, string? statusDescription, CancellationToken cancellationToken) + { + return _socket?.CloseAsync(closeStatus, statusDescription, cancellationToken) ?? Task.CompletedTask; + } + ///<inheritdoc/> + public override Task CloseOutputAsync(WebSocketCloseStatus closeStatus, string? statusDescription, CancellationToken cancellationToken) + { + if (_socket?.State == WebSocketState.Open || _socket?.State == WebSocketState.CloseSent) + { + return _socket.CloseOutputAsync(closeStatus, statusDescription, cancellationToken); + } + return Task.CompletedTask; + } + ///<inheritdoc/> + public override ValueTask<ValueWebSocketReceiveResult> ReceiveAsync(Memory<byte> buffer, CancellationToken cancellationToken) + { + _ = _socket ?? throw new WebSocketException(WebSocketError.ConnectionClosedPrematurely, "The connected socket has been disconnected"); + + return _socket.ReceiveAsync(buffer, cancellationToken); + } + ///<inheritdoc/> + public override Task<WebSocketReceiveResult> ReceiveAsync(ArraySegment<byte> buffer, CancellationToken cancellationToken) + { + _ = _socket ?? throw new WebSocketException(WebSocketError.ConnectionClosedPrematurely, "The connected socket has been disconnected"); + + return _socket.ReceiveAsync(buffer, cancellationToken); + } + ///<inheritdoc/> + public override ValueTask SendAsync(ReadOnlyMemory<byte> buffer, WebSocketMessageType messageType, bool endOfMessage, CancellationToken cancellationToken) + { + _ = _socket ?? throw new WebSocketException(WebSocketError.ConnectionClosedPrematurely, "The connected socket has been disconnected"); + return _socket.SendAsync(buffer, messageType, endOfMessage, cancellationToken); + } + ///<inheritdoc/> + public override Task SendAsync(ArraySegment<byte> buffer, WebSocketMessageType messageType, bool endOfMessage, CancellationToken cancellationToken) + { + _ = _socket ?? throw new WebSocketException(WebSocketError.ConnectionClosedPrematurely, "The connected socket has been disconnected"); + return _socket.SendAsync(buffer, messageType, endOfMessage, cancellationToken); + } + + ///<inheritdoc/> + public override void Dispose() + { + //Free buffer + _dataBuffer.Dispose(); + _socket?.Dispose(); + GC.SuppressFinalize(this); + } + } +} diff --git a/Net.Messaging.FBM/src/Client/README.md b/Net.Messaging.FBM/src/Client/README.md new file mode 100644 index 0000000..5aa8e76 --- /dev/null +++ b/Net.Messaging.FBM/src/Client/README.md @@ -0,0 +1,169 @@ +# VNLib.Net.Messaging.FBM.Client + +Fixed Buffer Messaging Protocol client library. High performance statful messaging +protocol built on top of HTTP web-sockets. Low/no allocation, completely asynchronous +while providing a TPL API. This library provides a simple asynchronous request/response +architecture to web-sockets. This was initially designed to provide an alternative to +complete HTTP request/response overhead, but allow a simple control flow for work +across a network. + +The base of the library relies on creating message objects that allocate fixed size +buffers are configured when the IFBMMessageis constructed. All data is written to the +internal buffer adhering to the format below. + +Messages consist of a 4 byte message id, a collection of headers, and a message body. +The first 4 bytes of a message is the ID (for normal messages a signed integer greater than 0), +0 is reserved for error conditions, and negative numbers are reserved for internal +messages. Headers are identified by a single byte, followed by a variable length UTF8 +encoded character sequence, followed by a termination of 0xFF, 0xF1 (may change). + +### Message structure + 4 byte positive (signed 32-bit integer) message id + 2 byte termination + 1 byte header-id + variable length UTF8 value + 2 byte termination + -- other headers -- + 2 byte termination (extra termination, ie: empty header) + variable length payload + (end of message is the end of the payload) + +Buffer sizes are generally negotiated on initial web-socket upgrade, so to buffer entire messages +in a single read/write from the web-socket. Received messages are read into memory until +the web-socket has no more data available. The message is then parsed and passed for processing +on the server side, or complete a pending request on the client side. Servers may drop the +web-socket connection and return an error if messages exceed the size of the pre-negotiated +buffer. Servers should validate buffer sizes before accepting a connection. + +This client library allows for messages to be streamed to the server, however this library +is optimized for fixed buffers, so streaming will not be the most efficient, and will likely +cause slow-downs in message transmission. However, since FBM relies on a streaming protocol, +so it was silly not to provide it. Streams add overhead of additional buffer allocation, +additional copy, and message fragmentation (multiple writes to the web-socket). Since frames +written to the web-socket must be synchronized, a mutex is held during transmission, which +means the more message overhead, the longer the blocking period on new messages. Mutex +acquisition will wait asynchronously when necessary. + +The goal of the FBM protocol for is to provide efficient use of resources (memory, network, +and minimize GC load) to transfer small messages truly asynchronously, at wire speeds, with +only web-socket and transport overhead. Using web-sockets simplifies implementation, and allows +comparability across platforms, languages, and versions. + +## fundamentals + +The main implementation is the FBMClient class. This class provides the means for creating +the stateful connection to the remote server. It also provides an internal FBMRequest message +rental (object cache) that created initialized FBMRequest messages. This class may be derrived +to provide additional functionality, such as handling control frames that may dynamically +alter the state of the connection (negotiation etc). A mechanism to do so is provided. + +### FBMClient layout + +``` + public class FBMClient : VnDisposeable, IStatefulConnection, ICacheHolder + { + //Raised when an error occurs during receiving or parsing + public event EventHandler<FMBClientErrorEventArgs>? ConnectionClosedOnError; + + //Raised when connection is closed, regardless of the cause + public event EventHandler? ConnectionClosed; + + //Connects to the remote server at the specified websocket address (ws:// or wss://) + public async Task ConnectAsync(Uri address, CancellationToken cancellation = default); + + //When connected, sends the specified message to the remote server + public async Task<FBMResponse> SendAsync(FBMRequest request, CancellationToken cancellationToken = default); + + //When connected, streams a message to the remote server, * the message payload must not be configured * + public async Task<FBMResponse> StreamDataAsync(FBMRequest request, Stream payload, ContentType ct, CancellationToken cancellationToken = default); + + //Disconnects from the remote server + public async Task DisconnectAsync(CancellationToken cancellationToken = default); + + //Releases all held resourses + public void Dispose(); //Inherrited from VnDisposeable + + ICacheHolder.CacheClear(); //Inherited member, clears cached FBMRequest objects + ICacheHolder.CacheHardClear(); //Inherited member, clears cached FBMRequest objects + } +``` + +### Example usage +``` + FBMClientConfig config = new() + { + //The size (in bytes) of the internal buffer to use when receiving messages from the server + RecvBufferSize = 1024, + + //FBMRequest buffer size (expected size of buffers, required for negotiation) + RequestBufferSize = 1024, + + //The size (in chars) of headers the FBMResponse should expect to buffer from the server + ResponseHeaderBufSize = 1024, + + //The absolute maximum message size to buffer from the server + MaxMessageSize = 10 * 1024 * 1024, //10KiB + + //The unmanaged heap the allocate buffers from + BufferHeap = Memory.Shared, + + //Web-socket keepalive frame interval + KeepAliveInterval = TimeSpan.FromSeconds(30), + + //Web-socket sub-protocol header value + SubProtocol = null + }; + + //Craete client from the config + using (FBMClient client = new(config)) + { + //Usually set some type of authentication headers before connecting + + /* + client.ClientSocket.SetHeader("Authorization", "Authorization token"); + */ + + //Connect to server + Uri address = new Uri("wss://localhost:8080/some/fbm/endpoint"); + await client.ConnectAsync(address, CancellationToken.None); + + do + { + //Rent request message + FBMRequest request = client.RentRequest(); + //Some arbitrary header value (or preconfigured header) + request.WriteHeader(0x10, "Hello"); + //Some arbitrary payload + request.WriteBody(new byte[] { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A }, ContentType.Binary); + //Send request message + using (FBMResponse response = await client.SendAsync(request, CancellationToken.None)) + { + //Extension method to raise exception if an invalid response was received (also use the response.IsSet flag) + response.ThrowIfNotSet(); + + //Check headers (using Linq to get first header) + string header1 = response.Headers.First().Value.ToString(); + + //Get payload, data is valid until the response is disposed + ReadOnlySpan<byte> body = response.ResponseBody; + } + //Return request + client.ReturnRequest(request); + //request.Dispose(); //Alternativly dispose message + + await Task.Delay(1000); + } + while(true); + } +``` + +## Final Notes + +XML Documentation is or will be provided for almost all public exports. APIs are intended to +be sensibly public and immutable to allow for easy extensability (via extension methods). I +often use extension libraries to provide additional functionality. (See cache library) + +This library is likely a niche use case, and is probably not for everyone. Unless you care +about reasonably efficient high frequency request/response messaging, this probably isnt +for you. This library provides a reasonable building block for distributed lock mechanisms +and small data caching.
\ No newline at end of file diff --git a/Net.Messaging.FBM/src/Exceptions/FBMException.cs b/Net.Messaging.FBM/src/Exceptions/FBMException.cs new file mode 100644 index 0000000..1d5c7db --- /dev/null +++ b/Net.Messaging.FBM/src/Exceptions/FBMException.cs @@ -0,0 +1,52 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMException.cs +* +* FBMException.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// A base exception class for all FBM Library exceptions + /// </summary> + public class FBMException : Exception + { + ///<inheritdoc/> + public FBMException() + { + } + ///<inheritdoc/> + public FBMException(string message) : base(message) + { + } + ///<inheritdoc/> + public FBMException(string message, Exception innerException) : base(message, innerException) + { + } + ///<inheritdoc/> + protected FBMException(SerializationInfo info, StreamingContext context) : base(info, context) + { + } + } +} diff --git a/Net.Messaging.FBM/src/Exceptions/FBMInvalidRequestException.cs b/Net.Messaging.FBM/src/Exceptions/FBMInvalidRequestException.cs new file mode 100644 index 0000000..ae42797 --- /dev/null +++ b/Net.Messaging.FBM/src/Exceptions/FBMInvalidRequestException.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMInvalidRequestException.cs +* +* FBMInvalidRequestException.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// Raised when a request message is not in a valid state and cannot be sent + /// </summary> + public class FBMInvalidRequestException : FBMException + { + public FBMInvalidRequestException() + { + } + + public FBMInvalidRequestException(string message) : base(message) + { + } + + public FBMInvalidRequestException(string message, Exception innerException) : base(message, innerException) + { + } + } +} diff --git a/Net.Messaging.FBM/src/Exceptions/InvalidResponseException.cs b/Net.Messaging.FBM/src/Exceptions/InvalidResponseException.cs new file mode 100644 index 0000000..3f0b970 --- /dev/null +++ b/Net.Messaging.FBM/src/Exceptions/InvalidResponseException.cs @@ -0,0 +1,52 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: InvalidResponseException.cs +* +* InvalidResponseException.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// Raised when a response to an FBM request is not in a valid state + /// </summary> + public class InvalidResponseException : FBMException + { + ///<inheritdoc/> + public InvalidResponseException() + { + } + ///<inheritdoc/> + public InvalidResponseException(string message) : base(message) + { + } + ///<inheritdoc/> + public InvalidResponseException(string message, Exception innerException) : base(message, innerException) + { + } + ///<inheritdoc/> + protected InvalidResponseException(SerializationInfo info, StreamingContext context) : base(info, context) + { + } + } +} diff --git a/Net.Messaging.FBM/src/Server/FBMContext.cs b/Net.Messaging.FBM/src/Server/FBMContext.cs new file mode 100644 index 0000000..fb39d1b --- /dev/null +++ b/Net.Messaging.FBM/src/Server/FBMContext.cs @@ -0,0 +1,85 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMContext.cs +* +* FBMContext.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Text; + +using VNLib.Utils.IO; +using VNLib.Utils.Memory.Caching; + +namespace VNLib.Net.Messaging.FBM.Server +{ + /// <summary> + /// A request/response pair message context + /// </summary> + public sealed class FBMContext : IReusable + { + private readonly Encoding _headerEncoding; + + /// <summary> + /// The request message to process + /// </summary> + public FBMRequestMessage Request { get; } + /// <summary> + /// The response message + /// </summary> + public FBMResponseMessage Response { get; } + /// <summary> + /// Creates a new reusable <see cref="FBMContext"/> + /// for use within a <see cref="ObjectRental{T}"/> + /// cache + /// </summary> + /// <param name="requestHeaderBufferSize">The size in characters of the request header buffer</param> + /// <param name="responseBufferSize">The size in characters of the response header buffer</param> + /// <param name="headerEncoding">The message header encoding instance</param> + public FBMContext(int requestHeaderBufferSize, int responseBufferSize, Encoding headerEncoding) + { + Request = new(requestHeaderBufferSize); + Response = new(responseBufferSize, headerEncoding); + _headerEncoding = headerEncoding; + } + + /// <summary> + /// Initializes the context with the buffered request data + /// </summary> + /// <param name="requestData">The request data buffer positioned at the begining of the request data</param> + /// <param name="connectionId">The unique id of the connection</param> + internal void Prepare(VnMemoryStream requestData, string connectionId) + { + Request.Prepare(requestData, connectionId, _headerEncoding); + //Message id is set after the request parses the incoming message + Response.Prepare(Request.MessageId); + } + + void IReusable.Prepare() + { + (Request as IReusable).Prepare(); + (Response as IReusable).Prepare(); + } + + bool IReusable.Release() + { + return (Request as IReusable).Release() & (Response as IReusable).Release(); + } + } +} diff --git a/Net.Messaging.FBM/src/Server/FBMListener.cs b/Net.Messaging.FBM/src/Server/FBMListener.cs new file mode 100644 index 0000000..1d50953 --- /dev/null +++ b/Net.Messaging.FBM/src/Server/FBMListener.cs @@ -0,0 +1,389 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMListener.cs +* +* FBMListener.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Threading; +using System.Net.WebSockets; +using System.Threading.Tasks; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Async; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; +using VNLib.Utils.Memory.Caching; +using VNLib.Plugins.Essentials; + +namespace VNLib.Net.Messaging.FBM.Server +{ + + /// <summary> + /// Method delegate for processing FBM messages from an <see cref="FBMListener"/> + /// when messages are received + /// </summary> + /// <param name="context">The message/connection context</param> + /// <param name="userState">The state parameter passed on client connected</param> + /// <param name="cancellationToken">A token that reflects the state of the listener</param> + /// <returns>A <see cref="Task"/> that resolves when processing is complete</returns> + public delegate Task RequestHandler(FBMContext context, object? userState, CancellationToken cancellationToken); + + /// <summary> + /// A FBM protocol listener. Listens for messages on a <see cref="WebSocketSession"/> + /// and raises events on requests. + /// </summary> + public class FBMListener + { + class ListeningSession + { + private readonly ReusableStore<FBMContext> CtxStore; + private readonly CancellationTokenSource Cancellation; + private readonly CancellationTokenRegistration Registration; + private readonly FBMListenerSessionParams Params; + + + public readonly object? UserState; + + public readonly SemaphoreSlim ResponseLock; + + public readonly WebSocketSession Socket; + + public readonly RequestHandler OnRecieved; + + public CancellationToken CancellationToken => Cancellation.Token; + + + public ListeningSession(WebSocketSession session, RequestHandler onRecieved, in FBMListenerSessionParams args, object? userState) + { + Params = args; + Socket = session; + UserState = userState; + OnRecieved = onRecieved; + + //Create cancellation and register for session close + Cancellation = new(); + Registration = session.Token.Register(Cancellation.Cancel); + + + ResponseLock = new(1); + CtxStore = ObjectRental.CreateReusable(ContextCtor); + } + + private FBMContext ContextCtor() => new(Params.MaxHeaderBufferSize, Params.ResponseBufferSize, Params.HeaderEncoding); + + /// <summary> + /// Cancels any pending opreations relating to the current session + /// </summary> + public void CancelSession() + { + Cancellation.Cancel(); + + //If dispose happens without any outstanding requests, we can dispose the session + if (_counter == 0) + { + CleanupInternal(); + } + } + + private void CleanupInternal() + { + Registration.Dispose(); + CtxStore.Dispose(); + Cancellation.Dispose(); + ResponseLock.Dispose(); + } + + + private uint _counter; + + /// <summary> + /// Rents a new <see cref="FBMContext"/> instance from the pool + /// and increments the counter + /// </summary> + /// <returns>The rented instance</returns> + /// <exception cref="ObjectDisposedException"></exception> + public FBMContext RentContext() + { + + if (Cancellation.IsCancellationRequested) + { + throw new ObjectDisposedException("The instance has been disposed"); + } + + //Rent context + FBMContext ctx = CtxStore.Rent(); + //Increment counter + Interlocked.Increment(ref _counter); + + return ctx; + } + + /// <summary> + /// Returns a previously rented context to the pool + /// and decrements the counter. If the session has been + /// cancelled, when the counter reaches 0, cleanup occurs + /// </summary> + /// <param name="ctx">The context to return</param> + public void ReturnContext(FBMContext ctx) + { + //Return the context + CtxStore.Return(ctx); + + uint current = Interlocked.Decrement(ref _counter); + + //No more contexts in use, dispose internals + if (Cancellation.IsCancellationRequested && current == 0) + { + ResponseLock.Dispose(); + Cancellation.Dispose(); + CtxStore.Dispose(); + } + } + } + + public const int SEND_SEMAPHORE_TIMEOUT_MS = 10 * 1000; + + private readonly IUnmangedHeap Heap; + + /// <summary> + /// Raised when a response processing error occured + /// </summary> + public event EventHandler<Exception>? OnProcessError; + + /// <summary> + /// Creates a new <see cref="FBMListener"/> instance ready for + /// processing connections + /// </summary> + /// <param name="heap">The heap to alloc buffers from</param> + public FBMListener(IUnmangedHeap heap) + { + Heap = heap; + } + + /// <summary> + /// Begins listening for requests on the current websocket until + /// a close message is received or an error occurs + /// </summary> + /// <param name="wss">The <see cref="WebSocketSession"/> to receive messages on</param> + /// <param name="handler">The callback method to handle incoming requests</param> + /// <param name="args">The arguments used to configured this listening session</param> + /// <param name="userState">A state parameter</param> + /// <returns>A <see cref="Task"/> that completes when the connection closes</returns> + public async Task ListenAsync(WebSocketSession wss, RequestHandler handler, FBMListenerSessionParams args, object? userState) + { + ListeningSession session = new(wss, handler, args, userState); + //Alloc a recieve buffer + using IMemoryOwner<byte> recvBuffer = Heap.DirectAlloc<byte>(args.RecvBufferSize); + + //Init new queue for dispatching work + AsyncQueue<VnMemoryStream> workQueue = new(true, true); + + //Start a task to process the queue + Task queueWorker = QueueWorkerDoWork(workQueue, session); + + try + { + //Listen for incoming messages + while (true) + { + //Receive a message + ValueWebSocketReceiveResult result = await wss.ReceiveAsync(recvBuffer.Memory); + //If a close message has been received, we can gracefully exit + if (result.MessageType == WebSocketMessageType.Close) + { + //Return close message + await wss.CloseSocketAsync(WebSocketCloseStatus.NormalClosure, "Goodbye"); + //break listen loop + break; + } + //create buffer for storing data + VnMemoryStream request = new(Heap); + //Copy initial data + request.Write(recvBuffer.Memory.Span[..result.Count]); + //Streaming read + while (!result.EndOfMessage) + { + //Read more data + result = await wss.ReceiveAsync(recvBuffer.Memory); + //Make sure the request is small enough to buffer + if (request.Length + result.Count > args.MaxMessageSize) + { + //dispose the buffer + request.Dispose(); + //close the socket with a message too big + await wss.CloseSocketAsync(WebSocketCloseStatus.MessageTooBig, "Buffer space exceeded for message. Goodbye"); + //break listen loop + goto Exit; + } + //write to buffer + request.Write(recvBuffer.Memory.Span[..result.Count]); + } + //Make sure data is available + if (request.Length == 0) + { + request.Dispose(); + continue; + } + //reset buffer position + _ = request.Seek(0, SeekOrigin.Begin); + //Enqueue the request + await workQueue.EnqueueAsync(request); + } + + Exit: + ; + } + finally + { + session.CancelSession(); + await queueWorker.ConfigureAwait(false); + } + } + + private async Task QueueWorkerDoWork(AsyncQueue<VnMemoryStream> queue, ListeningSession session) + { + try + { + while (true) + { + //Get work from queue + VnMemoryStream request = await queue.DequeueAsync(session.CancellationToken); + //Process request without waiting + _ = ProcessAsync(request, session).ConfigureAwait(false); + } + } + catch (OperationCanceledException) + { } + finally + { + //Cleanup any queued requests + while (queue.TryDequeue(out VnMemoryStream? stream)) + { + stream.Dispose(); + } + } + } + + private async Task ProcessAsync(VnMemoryStream data, ListeningSession session) + { + //Rent a new request object + FBMContext context = session.RentContext(); + try + { + //Prepare the request/response + context.Prepare(data, session.Socket.SocketID); + + if ((context.Request.ParseStatus & HeaderParseError.InvalidId) > 0) + { + OnProcessError?.Invoke(this, new FBMException($"Invalid messageid {context.Request.MessageId}, message length {data.Length}")); + return; + } + + //Check parse status flags + if ((context.Request.ParseStatus & HeaderParseError.HeaderOutOfMem) > 0) + { + OnProcessError?.Invoke(this, new OutOfMemoryException("Packet received with not enough space to store headers")); + } + //Determine if request is an out-of-band message + else if (context.Request.MessageId == Helpers.CONTROL_FRAME_MID) + { + //Process control frame + await ProcessOOBAsync(context); + } + else + { + //Invoke normal message handler + await session.OnRecieved.Invoke(context, session.UserState, session.CancellationToken); + } + + //Get response data + await using IAsyncMessageReader messageEnumerator = await context.Response.GetResponseDataAsync(session.CancellationToken); + + //Load inital segment + if (await messageEnumerator.MoveNextAsync() && !session.CancellationToken.IsCancellationRequested) + { + ValueTask sendTask; + + //Syncrhonize access to send data because we may need to stream data to the client + await session.ResponseLock.WaitAsync(SEND_SEMAPHORE_TIMEOUT_MS); + try + { + do + { + bool eof = !messageEnumerator.DataRemaining; + + //Send first segment + sendTask = session.Socket.SendAsync(messageEnumerator.Current, WebSocketMessageType.Binary, eof); + + /* + * WARNING! + * this code relies on the managed websocket impl that the websocket will read + * the entire buffer before returning. If this is not the case, this code will + * overwrite the memory buffer on the next call to move next. + */ + + //Move to next segment + if (!await messageEnumerator.MoveNextAsync()) + { + break; + } + + //Await previous send + await sendTask; + + } while (true); + } + finally + { + //release semaphore + session.ResponseLock.Release(); + } + + await sendTask; + } + + //No data to send + } + catch (Exception ex) + { + OnProcessError?.Invoke(this, ex); + } + finally + { + session.ReturnContext(context); + } + } + + /// <summary> + /// Processes an out-of-band request message (internal communications) + /// </summary> + /// <param name="outOfBandContext">The <see cref="FBMContext"/> containing the OOB message</param> + /// <returns>A <see cref="Task"/> that completes when the operation completes</returns> + protected virtual Task ProcessOOBAsync(FBMContext outOfBandContext) + { + return Task.CompletedTask; + } + } +} diff --git a/Net.Messaging.FBM/src/Server/FBMListenerBase.cs b/Net.Messaging.FBM/src/Server/FBMListenerBase.cs new file mode 100644 index 0000000..3e9fde2 --- /dev/null +++ b/Net.Messaging.FBM/src/Server/FBMListenerBase.cs @@ -0,0 +1,113 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMListenerBase.cs +* +* FBMListenerBase.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Logging; +using VNLib.Utils.Memory; +using VNLib.Plugins.Essentials; + +namespace VNLib.Net.Messaging.FBM.Server +{ + /// <summary> + /// Provides a simple base class for an <see cref="FBMListener"/> + /// processor + /// </summary> + public abstract class FBMListenerBase + { + + /// <summary> + /// The initialzied listener + /// </summary> + protected FBMListener? Listener { get; private set; } + /// <summary> + /// A provider to write log information to + /// </summary> + protected abstract ILogProvider Log { get; } + + /// <summary> + /// Initializes the <see cref="FBMListener"/> + /// </summary> + /// <param name="heap">The heap to alloc buffers from</param> + protected void InitListener(IUnmangedHeap heap) + { + Listener = new(heap); + //Attach service handler + Listener.OnProcessError += Listener_OnProcessError; + } + + /// <summary> + /// A single event service routine for servicing errors that occur within + /// the listener loop + /// </summary> + /// <param name="sender"></param> + /// <param name="e">The exception that was raised</param> + protected virtual void Listener_OnProcessError(object? sender, Exception e) + { + //Write the error to the log + Log.Error(e); + } + + private async Task OnReceivedAsync(FBMContext context, object? userState, CancellationToken token) + { + try + { + await ProcessAsync(context, userState, token); + } + catch (OperationCanceledException) + { + Log.Debug("Async operation cancelled"); + } + catch(Exception ex) + { + Log.Error(ex); + } + } + + /// <summary> + /// Begins listening for requests on the current websocket until + /// a close message is received or an error occurs + /// </summary> + /// <param name="wss">The <see cref="WebSocketSession"/> to receive messages on</param> + /// <param name="args">The arguments used to configured this listening session</param> + /// <param name="userState">A state token to use for processing events for this connection</param> + /// <returns>A <see cref="Task"/> that completes when the connection closes</returns> + public virtual async Task ListenAsync(WebSocketSession wss, FBMListenerSessionParams args, object? userState) + { + _ = Listener ?? throw new InvalidOperationException("The listener has not been intialized"); + await Listener.ListenAsync(wss, OnReceivedAsync, args, userState); + } + + /// <summary> + /// A method to service an incoming message + /// </summary> + /// <param name="context">The context containing the message to be serviced</param> + /// <param name="userState">A state token passed on client connected</param> + /// <param name="exitToken">A token that reflects the state of the listener</param> + /// <returns>A task that completes when the message has been serviced</returns> + protected abstract Task ProcessAsync(FBMContext context, object? userState, CancellationToken exitToken); + } +} diff --git a/Net.Messaging.FBM/src/Server/FBMListenerSessionParams.cs b/Net.Messaging.FBM/src/Server/FBMListenerSessionParams.cs new file mode 100644 index 0000000..c327475 --- /dev/null +++ b/Net.Messaging.FBM/src/Server/FBMListenerSessionParams.cs @@ -0,0 +1,62 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMListenerSessionParams.cs +* +* FBMListenerSessionParams.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Text; + +namespace VNLib.Net.Messaging.FBM.Server +{ + /// <summary> + /// Represents a configuration structure for an <see cref="FBMListener"/> + /// listening session + /// </summary> + public readonly struct FBMListenerSessionParams + { + /// <summary> + /// The size of the buffer to use while reading data from the websocket + /// in the listener loop + /// </summary> + public readonly int RecvBufferSize { get; init; } + /// <summary> + /// The size of the character buffer to store FBMheader values in + /// the <see cref="FBMRequestMessage"/> + /// </summary> + public readonly int MaxHeaderBufferSize { get; init; } + /// <summary> + /// The size of the internal message response buffer when + /// not streaming + /// </summary> + public readonly int ResponseBufferSize { get; init; } + /// <summary> + /// The FMB message header character encoding + /// </summary> + public readonly Encoding HeaderEncoding { get; init; } + + /// <summary> + /// The absolute maxium size (in bytes) message to process before + /// closing the websocket connection. This value should be negotiaed + /// by clients or hard-coded to avoid connection issues + /// </summary> + public readonly int MaxMessageSize { get; init; } + } +} diff --git a/Net.Messaging.FBM/src/Server/FBMRequestMessage.cs b/Net.Messaging.FBM/src/Server/FBMRequestMessage.cs new file mode 100644 index 0000000..ed36571 --- /dev/null +++ b/Net.Messaging.FBM/src/Server/FBMRequestMessage.cs @@ -0,0 +1,196 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMRequestMessage.cs +* +* FBMRequestMessage.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Buffers; +using System.Text.Json; +using System.Collections.Generic; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; +using VNLib.Utils.Memory.Caching; + +namespace VNLib.Net.Messaging.FBM.Server +{ + /// <summary> + /// Represents a client request message to be serviced + /// </summary> + public sealed class FBMRequestMessage : IReusable + { + private readonly List<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>> _headers; + private readonly int HeaderCharBufferSize; + /// <summary> + /// Creates a new resusable <see cref="FBMRequestMessage"/> + /// </summary> + /// <param name="headerBufferSize">The size of the buffer to alloc during initialization</param> + internal FBMRequestMessage(int headerBufferSize) + { + HeaderCharBufferSize = headerBufferSize; + _headers = new(); + } + + private char[]? _headerBuffer; + + /// <summary> + /// The ID of the current message + /// </summary> + public int MessageId { get; private set; } + /// <summary> + /// Gets the underlying socket-id fot the current connection + /// </summary> + public string? ConnectionId { get; private set; } + /// <summary> + /// The raw request message, positioned to the body section of the message data + /// </summary> + public VnMemoryStream? RequestBody { get; private set; } + /// <summary> + /// A collection of headers for the current request + /// </summary> + public IReadOnlyList<KeyValuePair<HeaderCommand, ReadOnlyMemory<char>>> Headers => _headers; + /// <summary> + /// Status flags set during the message parsing + /// </summary> + public HeaderParseError ParseStatus { get; private set; } + /// <summary> + /// The message body data as a <see cref="ReadOnlySpan{T}"/> + /// </summary> + public ReadOnlySpan<byte> BodyData => Helpers.GetRemainingData(RequestBody!); + + /// <summary> + /// Determines if the current message is considered a control frame + /// </summary> + public bool IsControlFrame { get; private set; } + + /// <summary> + /// Prepares the request to be serviced + /// </summary> + /// <param name="vms">The request data packet</param> + /// <param name="socketId">The unique id of the connection</param> + /// <param name="dataEncoding">The data encoding used to decode header values</param> + internal void Prepare(VnMemoryStream vms, string socketId, Encoding dataEncoding) + { + //Store request body + RequestBody = vms; + //Store message id + MessageId = Helpers.GetMessageId(Helpers.ReadLine(vms)); + //Check mid for control frame + if(MessageId == Helpers.CONTROL_FRAME_MID) + { + IsControlFrame = true; + } + else if (MessageId < 1) + { + ParseStatus |= HeaderParseError.InvalidId; + return; + } + + ConnectionId = socketId; + + //sliding window over remaining data from internal buffer + ForwardOnlyMemoryWriter<char> writer = new(_headerBuffer); + + //Accumulate headers + while (true) + { + //Read the next line from the current stream + ReadOnlySpan<byte> line = Helpers.ReadLine(vms); + if (line.IsEmpty) + { + //Done reading headers + break; + } + HeaderCommand cmd = Helpers.GetHeaderCommand(line); + //Get header value + ERRNO charsRead = Helpers.GetHeaderValue(line, writer.Remaining.Span, dataEncoding); + if (charsRead < 0) + { + //Out of buffer space + ParseStatus |= HeaderParseError.HeaderOutOfMem; + break; + } + else if (!charsRead) + { + //Invalid header + ParseStatus |= HeaderParseError.InvalidHeaderRead; + } + else + { + //Store header as a read-only sequence + _headers.Add(new(cmd, writer.Remaining[..(int)charsRead])); + //Shift buffer window + writer.Advance(charsRead); + } + } + } + + /// <summary> + /// Deserializes the request body into a new specified object type + /// </summary> + /// <typeparam name="T">The type of the object to deserialize</typeparam> + /// <param name="jso">The <see cref="JsonSerializerOptions"/> to use while deserializing data</param> + /// <returns>The deserialized object from the request body</returns> + /// <exception cref="JsonException"></exception> + public T? DeserializeBody<T>(JsonSerializerOptions? jso = default) + { + return BodyData.IsEmpty ? default : BodyData.AsJsonObject<T>(jso); + } + /// <summary> + /// Gets a <see cref="JsonDocument"/> of the request body + /// </summary> + /// <returns>The parsed <see cref="JsonDocument"/> if parsed successfully, or null otherwise</returns> + /// <exception cref="JsonException"></exception> + public JsonDocument? GetBodyAsJson() + { + Utf8JsonReader reader = new(BodyData); + return JsonDocument.TryParseValue(ref reader, out JsonDocument? jdoc) ? jdoc : default; + } + + void IReusable.Prepare() + { + ParseStatus = HeaderParseError.None; + //Alloc header buffer + _headerBuffer = ArrayPool<char>.Shared.Rent(HeaderCharBufferSize); + } + + + bool IReusable.Release() + { + //Dispose the request message + RequestBody?.Dispose(); + RequestBody = null; + //Clear headers before freeing buffer + _headers.Clear(); + //Free header-buffer + ArrayPool<char>.Shared.Return(_headerBuffer!); + _headerBuffer = null; + ConnectionId = null; + MessageId = 0; + IsControlFrame = false; + return true; + } + } +} diff --git a/Net.Messaging.FBM/src/Server/FBMResponseMessage.cs b/Net.Messaging.FBM/src/Server/FBMResponseMessage.cs new file mode 100644 index 0000000..1536c99 --- /dev/null +++ b/Net.Messaging.FBM/src/Server/FBMResponseMessage.cs @@ -0,0 +1,226 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: FBMResponseMessage.cs +* +* FBMResponseMessage.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Net.Http; +using VNLib.Utils.IO; +using VNLib.Utils.Extensions; +using VNLib.Utils.Memory.Caching; +using VNLib.Net.Messaging.FBM.Client; + +namespace VNLib.Net.Messaging.FBM.Server +{ + + /// <summary> + /// Represents an FBM request response container. + /// </summary> + public sealed class FBMResponseMessage : IReusable, IFBMMessage + { + internal FBMResponseMessage(int internalBufferSize, Encoding headerEncoding) + { + _headerAccumulator = new HeaderDataAccumulator(internalBufferSize); + _headerEncoding = headerEncoding; + _messageEnumerator = new(this); + } + + private readonly MessageSegmentEnumerator _messageEnumerator; + private readonly ISlindingWindowBuffer<byte> _headerAccumulator; + private readonly Encoding _headerEncoding; + + private IAsyncMessageBody? _messageBody; + + ///<inheritdoc/> + public int MessageId { get; private set; } + + void IReusable.Prepare() + { + (_headerAccumulator as HeaderDataAccumulator)!.Prepare(); + } + + bool IReusable.Release() + { + //Release header accumulator + _headerAccumulator.Close(); + + _messageBody = null; + + MessageId = 0; + + return true; + } + + /// <summary> + /// Initializes the response message with the specified message-id + /// to respond with + /// </summary> + /// <param name="messageId">The message id of the context to respond to</param> + internal void Prepare(int messageId) + { + //Reset accumulator when message id is written + _headerAccumulator.Reset(); + //Write the messageid to the begining of the headers buffer + MessageId = messageId; + _headerAccumulator.Append((byte)HeaderCommand.MessageId); + _headerAccumulator.Append(messageId); + _headerAccumulator.WriteTermination(); + } + + ///<inheritdoc/> + public void WriteHeader(HeaderCommand header, ReadOnlySpan<char> value) + { + WriteHeader((byte)header, value); + } + ///<inheritdoc/> + public void WriteHeader(byte header, ReadOnlySpan<char> value) + { + _headerAccumulator.WriteHeader(header, value, _headerEncoding); + } + + ///<inheritdoc/> + public void WriteBody(ReadOnlySpan<byte> body, ContentType contentType = ContentType.Binary) + { + //Append content type header + WriteHeader(HeaderCommand.ContentType, HttpHelpers.GetContentTypeString(contentType)); + //end header segment + _headerAccumulator.WriteTermination(); + //Write message body + _headerAccumulator.Append(body); + } + + /// <summary> + /// Sets the response message body + /// </summary> + /// <param name="messageBody">The <see cref="IAsyncMessageBody"/> to stream data from</param> + /// <exception cref="InvalidOperationException"></exception> + public void AddMessageBody(IAsyncMessageBody messageBody) + { + if(_messageBody != null) + { + throw new InvalidOperationException("The message body is already set"); + } + + //Append message content type header + WriteHeader(HeaderCommand.ContentType, HttpHelpers.GetContentTypeString(messageBody.ContentType)); + + //end header segment + _headerAccumulator.WriteTermination(); + + //Store message body + _messageBody = messageBody; + + } + + /// <summary> + /// Gets the internal message body enumerator and prepares the message for sending + /// </summary> + /// <param name="cancellationToken">A cancellation token</param> + /// <returns>A value task that returns the message body enumerator</returns> + internal async ValueTask<IAsyncMessageReader> GetResponseDataAsync(CancellationToken cancellationToken) + { + //try to buffer as much data in the header segment first + if(_messageBody?.RemainingSize > 0 && _headerAccumulator.RemainingSize > 0) + { + //Read data from the message + int read = await _messageBody.ReadAsync(_headerAccumulator.RemainingBuffer, cancellationToken); + //Advance accumulator to the read bytes + _headerAccumulator.Advance(read); + } + //return reusable enumerator + return _messageEnumerator; + } + + private class MessageSegmentEnumerator : IAsyncMessageReader + { + private readonly FBMResponseMessage _message; + + bool HeadersRead; + + public MessageSegmentEnumerator(FBMResponseMessage message) + { + _message = message; + } + + public ReadOnlyMemory<byte> Current { get; private set; } + + public bool DataRemaining { get; private set; } + + public async ValueTask<bool> MoveNextAsync() + { + //Attempt to read header segment first + if (!HeadersRead) + { + //Set the accumulated buffer + Current = _message._headerAccumulator.AccumulatedBuffer; + + //Update data remaining flag + DataRemaining = _message._messageBody?.RemainingSize > 0; + + //Set headers read flag + HeadersRead = true; + + return true; + } + else if (_message._messageBody?.RemainingSize > 0) + { + //Use the header buffer as the buffer for the message body + Memory<byte> buffer = _message._headerAccumulator.Buffer; + + //Read body segment + int read = await _message._messageBody.ReadAsync(buffer); + + //Update data remaining flag + DataRemaining = _message._messageBody.RemainingSize > 0; + + if (read > 0) + { + //Store the read segment + Current = buffer[..read]; + return true; + } + } + return false; + } + + public async ValueTask DisposeAsync() + { + //Clear current segment + Current = default; + + //Reset headers read flag + HeadersRead = false; + + //Dispose the message body if set + if (_message._messageBody != null) + { + await _message._messageBody.DisposeAsync(); + } + } + } + } + +} diff --git a/Net.Messaging.FBM/src/Server/HeaderDataAccumulator.cs b/Net.Messaging.FBM/src/Server/HeaderDataAccumulator.cs new file mode 100644 index 0000000..78b378d --- /dev/null +++ b/Net.Messaging.FBM/src/Server/HeaderDataAccumulator.cs @@ -0,0 +1,89 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: HeaderDataAccumulator.cs +* +* HeaderDataAccumulator.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; + +using VNLib.Utils.IO; + + +namespace VNLib.Net.Messaging.FBM.Server +{ + /// <summary> + /// Reusable sliding window impl + /// </summary> + internal class HeaderDataAccumulator : ISlindingWindowBuffer<byte> + { + private readonly int BufferSize; + + private byte[]? _memHandle; + + public HeaderDataAccumulator(int bufferSize) + { + BufferSize = bufferSize; + } + + ///<inheritdoc/> + public int WindowStartPos { get; private set; } + ///<inheritdoc/> + public int WindowEndPos { get; private set; } + ///<inheritdoc/> + public Memory<byte> Buffer => _memHandle.AsMemory(); + + ///<inheritdoc/> + public void Advance(int count) => WindowEndPos += count; + + ///<inheritdoc/> + public void AdvanceStart(int count) => WindowEndPos += count; + + ///<inheritdoc/> + public void Reset() + { + WindowStartPos = 0; + WindowEndPos = 0; + } + + /// <summary> + /// Allocates the internal message buffer + /// </summary> + public void Prepare() + { + _memHandle ??= ArrayPool<byte>.Shared.Rent(BufferSize); + } + + ///<inheritdoc/> + public void Close() + { + Reset(); + + if (_memHandle != null) + { + //Return the buffer to the pool + ArrayPool<byte>.Shared.Return(_memHandle); + _memHandle = null; + } + } + } + +} diff --git a/Net.Messaging.FBM/src/Server/IAsyncMessageBody.cs b/Net.Messaging.FBM/src/Server/IAsyncMessageBody.cs new file mode 100644 index 0000000..5566520 --- /dev/null +++ b/Net.Messaging.FBM/src/Server/IAsyncMessageBody.cs @@ -0,0 +1,57 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: IAsyncMessageBody.cs +* +* IAsyncMessageBody.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Net.Http; + +namespace VNLib.Net.Messaging.FBM +{ + /// <summary> + /// A disposable message body container for asynchronously reading a variable length message body + /// </summary> + public interface IAsyncMessageBody : IAsyncDisposable + { + /// <summary> + /// The message body content type + /// </summary> + ContentType ContentType { get; } + + /// <summary> + /// The number of bytes remaining to be read from the message body + /// </summary> + int RemainingSize { get; } + + /// <summary> + /// Reads the next chunk of data from the message body + /// </summary> + /// <param name="buffer">The buffer to copy output data to</param> + /// <param name="token">A token to cancel the operation</param> + /// <returns></returns> + ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken token = default); + } + +} diff --git a/Net.Messaging.FBM/src/Server/IAsyncMessageReader.cs b/Net.Messaging.FBM/src/Server/IAsyncMessageReader.cs new file mode 100644 index 0000000..b2abe8d --- /dev/null +++ b/Net.Messaging.FBM/src/Server/IAsyncMessageReader.cs @@ -0,0 +1,42 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Messaging.FBM +* File: IAsyncMessageReader.cs +* +* IAsyncMessageReader.cs is part of VNLib.Net.Messaging.FBM which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Messaging.FBM is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Messaging.FBM is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; + + +namespace VNLib.Net.Messaging.FBM.Server +{ + /// <summary> + /// Internal message body reader/enumerator for FBM messages + /// </summary> + internal interface IAsyncMessageReader : IAsyncEnumerator<ReadOnlyMemory<byte>> + { + /// <summary> + /// A value that indicates if there is data remaining after a + /// </summary> + bool DataRemaining { get; } + } + +} diff --git a/Net.Messaging.FBM/src/Server/readme.md b/Net.Messaging.FBM/src/Server/readme.md new file mode 100644 index 0000000..489e58f --- /dev/null +++ b/Net.Messaging.FBM/src/Server/readme.md @@ -0,0 +1,35 @@ +# VNLib.Net.Messaging.FBM.Server + +Fixed Buffer Messaging Protocol server library. High performance statful messaging +protocol built on top of HTTP web-sockets. Low/no allocation, completely asynchronous +while providing a TPL API. This library provides a simple asynchronous request/response +architecture to web-sockets. This was initially designed to provide an alternative to +complete HTTP request/response overhead, but allow a simple control flow for work +across a network. + +Messages consist of a 4 byte message id, a collection of headers, and a message body. +The first 4 bytes of a message is the ID (for normal messages a signed integer greater than 0), +0 is reserved for error conditions, and negative numbers are reserved for internal +messages. Headers are identified by a single byte, followed by a variable length UTF8 +encoded character sequence, followed by a termination of 0xFF, 0xF1 (may change). + +### Message structure + 4 byte positive (signed 32-bit integer) message id + 2 byte termination + 1 byte header-id + variable length UTF8 value + 2 byte termination + -- other headers -- + 2 byte termination (extra termination, ie: empty header) + variable length payload + (end of message is the end of the payload) + + +XML Documentation is or will be provided for almost all public exports. APIs are intended to +be sensibly public and immutable to allow for easy extensability (via extension methods). I +often use extension libraries to provide additional functionality. (See cache library) + +This library is likely a niche use case, and is probably not for everyone. Unless you care +about reasonably efficient high frequency request/response messaging, this probably isnt +for you. This library provides a reasonable building block for distributed lock mechanisms +and small data caching.
\ No newline at end of file diff --git a/Net.Messaging.FBM/src/VNLib.Net.Messaging.FBM.csproj b/Net.Messaging.FBM/src/VNLib.Net.Messaging.FBM.csproj new file mode 100644 index 0000000..9440e69 --- /dev/null +++ b/Net.Messaging.FBM/src/VNLib.Net.Messaging.FBM.csproj @@ -0,0 +1,33 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <Authors>Vaughn Nugent</Authors> + <Version>1.0.1.1</Version> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <Nullable>enable</Nullable> + <PackageProjectUrl>www.vaughnnugent.com/resources</PackageProjectUrl> + <AnalysisLevel>latest-all</AnalysisLevel> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Essentials\src\VNLib.Plugins.Essentials.csproj" /> + <ProjectReference Include="..\..\Http\src\VNLib.Net.Http.csproj" /> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + </ItemGroup> + +</Project> diff --git a/Net.Transport.SimpleTCP/LICENSE.txt b/Net.Transport.SimpleTCP/LICENSE.txt new file mode 100644 index 0000000..147bcd6 --- /dev/null +++ b/Net.Transport.SimpleTCP/LICENSE.txt @@ -0,0 +1,195 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU Affero GPL version 3.0 (or any later version). + +GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/> +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. +Preamble + +The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. + +A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. + +The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. + +An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. + +The precise terms and conditions for copying, distribution and modification follow. +TERMS AND CONDITIONS +0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public License. + +"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based on the Program. + +To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. + +An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. +1. Source Code. + +The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. + +A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same work. +2. Basic Permissions. + +All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. +3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. + +When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. +4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. +5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified it, and giving a relevant date. + b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". + c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. + d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. + +A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. +6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: + + a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. + b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. + c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. + d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. + e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. + +"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. + +If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). + +The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. + +Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. +7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or + b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or + c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or + d) Limiting the use for publicity purposes of names of licensors or authors of the material; or + e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or + f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. + +All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. +8. Termination. + +You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). + +However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. + +Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. + +Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. +9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. +10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. +11. Patents. + +A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. + +If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. + +A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. +12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. +13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. +14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. + +Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. +15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. +16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/README.md b/Net.Transport.SimpleTCP/README.md new file mode 100644 index 0000000..2db8775 --- /dev/null +++ b/Net.Transport.SimpleTCP/README.md @@ -0,0 +1,61 @@ +# VNLib.Net.Transport.SimpleTCP + +_A managed .NET simple, high performance - single process, low/no allocation, fully asynchronous, tcp socket server._ + +This library was created for use with the VNLib.Net.Http library and subsequent stacked framework libraries, however it was designed to be useful as a standalone high-performance .NET tcp listener. This library relies on the managed .NET [System.IO.Pipelines](https://github.com/dotnet/docs/blob/main/docs/standard/io/pipelines.md) library, and the **VNLib.Utils** library. + +##### SSL Support +The TcpServer manages ssl/tls using the SslStream class to make tls as transparent to the application as possible. The server manages authentication and negotiation based on the configured `SslServerAuthenticationOptions` + +## Usage + +```programming language C# + //Init config + TCPConfig config = new() + { + ... configure + } + + //Create the new server + TcpServer server = new(config); + + //Open the socket and begin listening for connections until the token is cancelled + server.Start(<cancellationToken>); + + //Listen for connections + while(true) + { + TransportEventContext ctx = await server.AcceptAsync(<cancellationToken>); + + try + { + ..Do stuff with context, such as read data from stream + byte[] buffer = new byte [1024]; + int count = await ctx.ConnectionStream,ReadAsync(buffer) + } + finally + { + await ctx.CloseConnectionAsync(); + } + } +``` + + +### Tuning information + +##### Internal buffers +Internal buffers are allocated for reading and writing to the internal socket. Receive buffers sizes are set to the `Socket.ReceiveBufferSize`, +so if you wish to reduce socket memory consumption, you may use the `TCPConfig.OnSocketCreated` callback method to configure your socket accordingly. + +##### Threading +This library uses the SocketAsyncEventArgs WinSock socket programming paradigm, so the `TPCConfig.AcceptThread` configuration property is the number of outstanding SocketAsyncEvents that will be pending. This value should be tuned to your use case, lower numbers relative to processor count may yield less accepts/second, higher numbers may see no increase or even reduced performance. + +##### Internal object cache +TcpServer maintains a complete object cache (VNLib.Utils.Memory.Caching.ObjectCache) which may grow quite large for your application depending on load, tuning the cache quota config property may be useful for your application. Lower numbers will increase GC load, higher values (or disabled) will likely yield a larger working set. Because of this the TcpServer class implements the ICacheHolder interface. **Note:** because TcpServer caches store disposable objects, the `CacheClear()` method does nothing. To programatically clear these caches, call the `CacheHardClear()` method. + +##### Memory pools +Since this library implements the System.IO.Pipelines, it uses the `MemoryPool<byte>` memory manager interface, you may consider using the VNLib.Utils `IUnmanagedHeap.ToPool<T>()` extension method to convert your `IUnmanagedHeap` to a `MemoryPool<byte>` + +## Lisence +The software in this repository is licensed under the GNU Affero General Public License (or any later version). +See the LICENSE files for more information.
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/src/ITransportInterface.cs b/Net.Transport.SimpleTCP/src/ITransportInterface.cs new file mode 100644 index 0000000..7d21995 --- /dev/null +++ b/Net.Transport.SimpleTCP/src/ITransportInterface.cs @@ -0,0 +1,85 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: ITransportInterface.cs +* +* ITransportInterface.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Net.Transport.Tcp +{ + /// <summary> + /// Abstraction layer for TCP transport operations with + /// sync and async support. + /// </summary> + interface ITransportInterface + { + /// <summary> + /// Gets or sets the read timeout in milliseconds + /// </summary> + int RecvTimeoutMs { get; set; } + + /// <summary> + /// Gets or set the time (in milliseconds) the transport should wait for a send operation + /// </summary> + int SendTimeoutMs { get; set; } + + /// <summary> + /// Performs an asynchronous send operation + /// </summary> + /// <param name="data">The buffer containing the data to send to the client</param> + /// <param name="cancellation">A token to cancel the operation</param> + /// <returns>A ValueTask that completes when the send operation is complete</returns> + ValueTask SendAsync(ReadOnlyMemory<byte> data, CancellationToken cancellation); + + /// <summary> + /// Performs an asynchronous send operation + /// </summary> + /// <param name="buffer">The data buffer to write received data to</param> + /// <param name="cancellation">A token to cancel the operation</param> + /// <returns>A ValueTask that returns the number of bytes read into the buffer</returns> + ValueTask<int> RecvAsync(Memory<byte> buffer, CancellationToken cancellation); + + /// <summary> + /// Performs a synchronous send operation + /// </summary> + /// <param name="data">The buffer to send to the client</param> + void Send(ReadOnlySpan<byte> data); + + /// <summary> + /// Performs a synchronous receive operation + /// </summary> + /// <param name="buffer">The buffer to copy output data to</param> + /// <returns>The number of bytes received</returns> + int Recv(Span<byte> buffer); + + /// <summary> + /// Raised when the interface is no longer required and resources + /// related to the connection should be released. + /// </summary> + /// <returns>A task that resolves when the operation is complete</returns> + Task CloseAsync(); + + } +}
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/src/ReusableNetworkStream.cs b/Net.Transport.SimpleTCP/src/ReusableNetworkStream.cs new file mode 100644 index 0000000..510419c --- /dev/null +++ b/Net.Transport.SimpleTCP/src/ReusableNetworkStream.cs @@ -0,0 +1,173 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: ReusableNetworkStream.cs +* +* ReusableNetworkStream.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +/* + * A special stream that sits betnween the socket/pipeline listener + * that marshals data between the application and the socket pipeline. + * This stream uses a timer to cancel recv events. Because of this and + * pipeline aspects, it supports full duplex IO but it is not thread safe. + * + * IE one thread can read and write, but not more + */ + + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Transport.Tcp +{ + + /// <summary> + /// A reusable stream that marshals data between the socket pipeline and the application + /// </summary> + internal class ReusableNetworkStream : Stream + { + #region stream basics + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override bool CanTimeout => true; + public override long Length => throw new NotSupportedException(); + public override long Position { get => throw new NotSupportedException(); set => throw new NotImplementedException(); } + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + public override void SetLength(long value) => throw new NotSupportedException(); + #endregion + + //Read timeout to use when receiving data + public override int ReadTimeout + { + get => Transport.RecvTimeoutMs; + set + { + if (value < 0) + { + throw new ArgumentException("Write timeout must be a 32bit signed integer larger than 0"); + } + Transport.RecvTimeoutMs = value; + } + } + + // Write timeout is not currently used, becasue the writer managed socket timeouts + public override int WriteTimeout + { + get => Transport.SendTimeoutMs; + set + { + if(value < 0) + { + throw new ArgumentException("Write timeout must be a 32bit signed integer larger than 0"); + } + Transport.SendTimeoutMs = value; + } + } + + //Timer used to cancel pipeline recv timeouts + private readonly ITransportInterface Transport; + + internal ReusableNetworkStream(ITransportInterface transport) + { + Transport = transport; + } + + ///<inheritdoc/> + public override void Close() + { } + ///<inheritdoc/> + public override Task FlushAsync(CancellationToken cancellationToken) + { + return Task.CompletedTask; + } + ///<inheritdoc/> + public override void Flush() + { + } + + ///<inheritdoc/> + public override int Read(byte[] buffer, int offset, int count) => Read(buffer.AsSpan(offset, count)); + ///<inheritdoc/> + public override int Read(Span<byte> buffer) + { + return Transport.Recv(buffer); + } + + ///<inheritdoc/> + public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + //Since read returns a value, it isnt any cheaper not to alloc a task around the value-task + return ReadAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + ///<inheritdoc/> + public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default) + { + return Transport.RecvAsync(buffer, cancellationToken); + } + + + ///<inheritdoc/> + public override void Write(byte[] buffer, int offset, int count) => Write(buffer.AsSpan(offset, count)); + ///<inheritdoc/> + public override void Write(ReadOnlySpan<byte> buffer) => Transport.Send(buffer); + + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + //Allow synchronous complete to avoid alloc + return WriteAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + + ///<inheritdoc/> + ///<exception cref="IOException"></exception> + ///<exception cref="ObjectDisposedException"></exception> + public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellation = default) + { + return Transport.SendAsync(buffer, cancellation); + } + + /* + * Override dispose to intercept base cleanup until the internal release + */ + /// <summary> + /// Not supported + /// </summary> + public new void Dispose() + { + //Call sync + Task closing = Transport.CloseAsync(); + closing.Wait(); + } + + public override ValueTask DisposeAsync() + { + return new ValueTask(Transport.CloseAsync()); + } + + public override Task CopyToAsync(Stream destination, int bufferSize, CancellationToken cancellationToken) + { + throw new NotSupportedException("CopyToAsync is not supported"); + } + } +}
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/src/SocketPipeLineWorker.cs b/Net.Transport.SimpleTCP/src/SocketPipeLineWorker.cs new file mode 100644 index 0000000..89c46e1 --- /dev/null +++ b/Net.Transport.SimpleTCP/src/SocketPipeLineWorker.cs @@ -0,0 +1,521 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: SocketPipeLineWorker.cs +* +* SocketPipeLineWorker.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Threading; +using System.Net.Sockets; +using System.IO.Pipelines; +using System.Threading.Tasks; + +using VNLib.Utils.Memory; +using VNLib.Utils.Memory.Caching; +using VNLib.Utils.Extensions; + +namespace VNLib.Net.Transport.Tcp +{ + /// <summary> + /// A reuseable socket pipeline provider, that marshals data from a network stream + /// to a connected socket. + /// </summary> + internal sealed class SocketPipeLineWorker : ITransportInterface, IReusable + { + public void Prepare() + {} + + public bool Release() + { + /* + * If the pipeline has been started, then the pipes + * will be completed by the worker threads (or by the streams) + * and when release is called, there will no longer be + * an observer for the result, which means the pipes + * may be safely reset for reuse + */ + if (_recvTask != null) + { + //Since the + SendPipe.Reset(); + RecvPipe.Reset(); + } + /* + * If socket had an error and was not started, + * it means there may be data written to the + * recv pipe from the accept operation, that + * needs to be cleared + */ + else + { + //Complete the recvpipe then reset it to discard buffered data + RecvPipe.Reader.Complete(); + RecvPipe.Writer.Complete(); + //now reset it + RecvPipe.Reset(); + } + + //Cleanup tasks + _recvTask = null; + _sendTask = null; + + //Cleanup cts + _cts?.Dispose(); + _cts = null; + + return true; + } + + private Task? _recvTask; + private Task? _sendTask; + + private CancellationTokenSource? _cts; + + public readonly ReusableNetworkStream NetworkStream; + + private readonly Pipe SendPipe; + private readonly Pipe RecvPipe; + private readonly Timer RecvTimer; + private readonly Timer SendTimer; + private readonly Stream RecvStream; + + public int SendTimeoutMs { get; set; } + + public int RecvTimeoutMs { get; set; } + + + /// <summary> + /// Initalizes a new reusable socket pipeline worker + /// </summary> + /// <param name="pipeOptions"></param> + public SocketPipeLineWorker(PipeOptions pipeOptions) + { + //Init pipes + SendPipe = new(pipeOptions); + RecvPipe = new(pipeOptions); + + RecvStream = RecvPipe.Reader.AsStream(true); + + //Init timers to infinite + RecvTimer = new(OnRecvTimerElapsed, state: this, Timeout.Infinite, Timeout.Infinite); + SendTimer = new(OnSendTimerElapsed, state: this, Timeout.Infinite, Timeout.Infinite); + + //Init reusable network stream + NetworkStream = new(this); + + SendTimeoutMs = Timeout.Infinite; + RecvTimeoutMs = Timeout.Infinite; + } + + /// <summary> + /// Gets a buffer used during a socket accept operation + /// </summary> + /// <param name="bufferSize">The size hint of the buffer to get</param> + /// <returns>A memory structure of the specified size</returns> + public Memory<byte> GetMemory(int bufferSize) => RecvPipe.Writer.GetMemory(bufferSize); + + /// <summary> + /// Begins async work to receive and send data on a connected socket + /// </summary> + /// <param name="client">The socket to read/write from</param> + /// <param name="bytesTransferred">The number of bytes to be commited</param> + public void Start(Socket client, int bytesTransferred) + { + //Advance writer + RecvPipe.Writer.Advance(bytesTransferred); + //begin recv tasks, and pass inital data to be flushed flag + _recvTask = RecvDoWorkAsync(client, bytesTransferred > 0); + _sendTask = SendDoWorkAsync(client); + } + + + /* + * NOTES + * + * Timers used to maintain resource exhuastion independent + * of the actual socket pipeline, so to preserve the state + * of the pipelines until the writer is closed. + * + * This choice was made to allow the api consumer to decide how to + * process a timeout without affecting the state of the pipelines + * or socket until the close event. + */ + + private void OnRecvTimerElapsed(object? state) + { + //cancel pending read on recv pipe when timout expires + RecvPipe.Reader.CancelPendingRead(); + } + + private void OnSendTimerElapsed(object? state) + { + //Cancel pending flush + SendPipe.Writer.CancelPendingFlush(); + } + + /* + * Pipeline worker tasks. Listen for data on the socket, + * and listen for data on the pipe to marshal data between + * the pipes and the socket + */ + + private async Task SendDoWorkAsync(Socket sock) + { + Exception? cause = null; + try + { + //Enter work loop + while (true) + { + //wait for data from the write pipe and write it to the socket + ReadResult result = await SendPipe.Reader.ReadAsync(); + //Catch error/cancel conditions and break the loop + if (result.IsCanceled || !sock.Connected || result.Buffer.IsEmpty) + { + break; + } + //get sequence + ReadOnlySequence<byte> buffer = result.Buffer; + + //Get enumerator to write memory segments + ReadOnlySequence<byte>.Enumerator enumerator = buffer.GetEnumerator(); + + //Begin enumerator + while (enumerator.MoveNext()) + { + + /* + * Using a foward only reader allows the following loop + * to track the ammount of data written to the socket + * until the entire segment has been sent or if it has + * move to the next segment + */ + + ForwardOnlyMemoryReader<byte> reader = new(enumerator.Current); + + while(reader.WindowSize > 0) + { + //Write segment to socket, and upate written data + int written = await sock.SendAsync(reader.Window, SocketFlags.None); + + if(written >= reader.WindowSize) + { + //All data was written + break; + } + + //Advance unread window to end of the written data + reader.Advance(written); + } + //Advance to next window/segment + } + + //Advance pipe + SendPipe.Reader.AdvanceTo(buffer.End); + + //Pipe has been completed and all data was written + if (result.IsCompleted) + { + break; + } + } + } + catch (Exception ex) + { + cause = ex; + } + finally + { + //Complete the send pipe writer + await SendPipe.Reader.CompleteAsync(cause); + + //Cancel the recv task + _cts!.Cancel(); + } + } + + private async Task RecvDoWorkAsync(Socket sock, bool initialData) + { + //init new cts + _cts = new(); + + Exception? cause = null; + try + { + //Avoid syscall? + int bufferSize = sock.ReceiveBufferSize; + + //If initial data was buffered, it needs to be published to the reader + if (initialData) + { + //Flush initial data + FlushResult res = await RecvPipe.Writer.FlushAsync(CancellationToken.None); + + if (res.IsCompleted || res.IsCanceled) + { + //Exit + return; + } + } + + //Enter work loop + while (true) + { + //Get buffer from pipe writer + Memory<byte> buffer = RecvPipe.Writer.GetMemory(bufferSize); + + //Wait for data or error from socket + int count = await sock.ReceiveAsync(buffer, SocketFlags.None, _cts.Token); + + //socket returned emtpy data + if (count == 0 || !sock.Connected) + { + break; + } + + //Advance/notify the pipe + RecvPipe.Writer.Advance(count); + + //Flush data at top of loop, since data is available from initial accept + FlushResult res = await RecvPipe.Writer.FlushAsync(CancellationToken.None); + + //Writing has completed, time to exit + if (res.IsCompleted || res.IsCanceled) + { + break; + } + } + } + //Normal exit + catch (OperationCanceledException) + {} + catch (SocketException se) + { + cause = se; + //Cancel sending reader task because the socket has an error and cannot be used + SendPipe.Reader.CancelPendingRead(); + } + catch (Exception ex) + { + cause = ex; + } + finally + { + //Stop timer incase exception + RecvTimer.Stop(); + + //Cleanup and complete the writer + await RecvPipe.Writer.CompleteAsync(cause); + //The recv reader is completed by the network stream + } + } + + /// <summary> + /// The internal cleanup/dispose method to be called + /// when the pipeline is no longer needed + /// </summary> + public void DisposeInternal() + { + RecvTimer.Dispose(); + SendTimer.Dispose(); + + //Perform some managed cleanup + + //Cleanup tasks + _recvTask = null; + _sendTask = null; + + //Cleanup cts + _cts?.Dispose(); + _cts = null; + } + + + private static async Task AwaitFlushTask(ValueTask<FlushResult> valueTask, Timer? sendTimer) + { + try + { + FlushResult result = await valueTask.ConfigureAwait(false); + + if (result.IsCanceled) + { + throw new OperationCanceledException("The operation was canceled by the underlying PipeWriter"); + } + } + finally + { + sendTimer?.Stop(); + } + } + + private ValueTask SendWithTimerInternalAsync(ReadOnlyMemory<byte> data, CancellationToken cancellation) + { + //Start send timer + SendTimer.Restart(SendTimeoutMs); + try + { + //Send the segment + ValueTask<FlushResult> result = SendPipe.Writer.WriteAsync(data, cancellation); + + //Task completed successfully, so + if (result.IsCompletedSuccessfully) + { + //Stop timer + SendTimer.Stop(); + + //Safe to get the rseult + FlushResult fr = result.Result; + //Check for canceled and throw + return fr.IsCanceled + ? throw new OperationCanceledException("The write operation was canceled by the underlying PipeWriter") + : ValueTask.CompletedTask; + } + else + { + //Wrap the task in a ValueTask since it must be awaited, and will happen on background thread + return new(AwaitFlushTask(result, SendTimer)); + } + } + catch + { + //Stop timer on exception + SendTimer.Stop(); + throw; + } + } + + private ValueTask SendWithoutTimerInternalAsync(ReadOnlyMemory<byte> data, CancellationToken cancellation) + { + //Send the segment + ValueTask<FlushResult> result = SendPipe.Writer.WriteAsync(data, cancellation); + + //Task completed successfully, so + if (result.IsCompletedSuccessfully) + { + //Safe to get the rseult + FlushResult fr = result.Result; + //Check for canceled and throw + return fr.IsCanceled + ? throw new OperationCanceledException("The write operation was canceled by the underlying PipeWriter") + : ValueTask.CompletedTask; + } + else + { + //Wrap the task in a ValueTask since it must be awaited, and will happen on background thread + return new(AwaitFlushTask(result, null)); + } + } + + ValueTask ITransportInterface.SendAsync(ReadOnlyMemory<byte> data, CancellationToken cancellation) + { + //Use timer if timeout is set, dont otherwise + return SendTimeoutMs < 1 ? SendWithoutTimerInternalAsync(data, cancellation) : SendWithTimerInternalAsync(data, cancellation); + } + + + void ITransportInterface.Send(ReadOnlySpan<byte> data) + { + //Determine if the send timer should be used + Timer? _timer = SendTimeoutMs < 1 ? null : SendTimer; + + //Write data directly to the writer buffer + SendPipe.Writer.Write(data); + + //Start send timer + _timer?.Restart(SendTimeoutMs); + + try + { + //Send the segment + ValueTask<FlushResult> result = SendPipe.Writer.FlushAsync(CancellationToken.None); + + //Task completed successfully, so + if (result.IsCompletedSuccessfully) + { + //Safe to get the rseult + FlushResult fr = result.Result; + + //Check for canceled and throw + if (fr.IsCanceled) + { + throw new OperationCanceledException("The write operation was canceled by the underlying PipeWriter"); + } + } + else + { + //Await the result + FlushResult fr = result.ConfigureAwait(false).GetAwaiter().GetResult(); + + if (fr.IsCanceled) + { + throw new OperationCanceledException("The write operation was canceled by the underlying PipeWriter"); + } + } + } + finally + { + //Stop timer + _timer?.Stop(); + } + } + + async ValueTask<int> ITransportInterface.RecvAsync(Memory<byte> buffer, CancellationToken cancellation) + { + //Restart timer + RecvTimer.Restart(RecvTimeoutMs); + try + { + return await RecvStream.ReadAsync(buffer, cancellation); + } + finally + { + RecvTimer.Stop(); + } + } + + int ITransportInterface.Recv(Span<byte> buffer) + { + //Restart timer + RecvTimer.Restart(RecvTimeoutMs); + try + { + return RecvStream.Read(buffer); + } + finally + { + RecvTimer.Stop(); + } + } + + Task ITransportInterface.CloseAsync() + { + //Complete the send pipe writer since stream is closed + ValueTask vt = SendPipe.Writer.CompleteAsync(); + //Complete the recv pipe reader since its no longer used + ValueTask rv = RecvPipe.Reader.CompleteAsync(); + //Join worker tasks, no alloc if completed sync, otherwise alloc anyway + return Task.WhenAll(vt.AsTask(), rv.AsTask(), _recvTask!, _sendTask!); + } + + } +} diff --git a/Net.Transport.SimpleTCP/src/TCPConfig.cs b/Net.Transport.SimpleTCP/src/TCPConfig.cs new file mode 100644 index 0000000..6955e63 --- /dev/null +++ b/Net.Transport.SimpleTCP/src/TCPConfig.cs @@ -0,0 +1,97 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: TCPConfig.cs +* +* TCPConfig.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Buffers; +using System.Net.Sockets; +using System.Net.Security; + +using VNLib.Utils.Logging; + +namespace VNLib.Net.Transport.Tcp +{ + /// <summary> + /// Represents the required configuration variables for the transport + /// </summary> + public readonly struct TCPConfig + { + /// <summary> + /// The <see cref="IPEndPoint"/> the listening socket will bind to + /// </summary> + public readonly IPEndPoint LocalEndPoint { get; init; } + /// <summary> + /// The log provider used to write logging information to + /// </summary> + public readonly ILogProvider Log { get; init; } + /// <summary> + /// If TCP keepalive is enabled, the amount of time the connection is considered alive before another probe message is sent + /// </summary> + public readonly int TcpKeepAliveTime { get; init; } + /// <summary> + /// If TCP keepalive is enabled, the amount of time the connection will wait for a keepalive message + /// </summary> + public readonly int KeepaliveInterval { get; init; } + /// <summary> + /// Enables TCP keepalive + /// </summary> + public readonly bool TcpKeepalive { get; init; } + /// <summary> + /// The authentication options to use for processing TLS connections. This value must be set when a certificate has been specified + /// </summary> + public readonly SslServerAuthenticationOptions? AuthenticationOptions { get; init; } + /// <summary> + /// The maximum number of waiting WSA asynchronous socket accept operations + /// </summary> + public readonly uint AcceptThreads { get; init; } + /// <summary> + /// The maximum size (in bytes) the transport will buffer in + /// the receiving pipeline. + /// </summary> + public readonly int MaxRecvBufferData { get; init; } + /// <summary> + /// The listener socket backlog count + /// </summary> + public readonly int BackLog { get; init; } + /// <summary> + /// The <see cref="MemoryPool{T}"/> to allocate transport buffers from + /// </summary> + public readonly MemoryPool<byte> BufferPool { get; init; } + /// <summary> + /// <para> + /// The maxium number of event objects that will be cached + /// during normal operation + /// </para> + /// <para> + /// WARNING: Setting this value too low will cause significant CPU overhead and GC load + /// </para> + /// </summary> + public readonly int CacheQuota { get; init; } + /// <summary> + /// An optional callback invoked after the socket has been created + /// for optional appliction specific socket configuration + /// </summary> + public Action<Socket>? OnSocketCreated { get; init; } + } +}
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/src/TcpServer.cs b/Net.Transport.SimpleTCP/src/TcpServer.cs new file mode 100644 index 0000000..fc0bcc5 --- /dev/null +++ b/Net.Transport.SimpleTCP/src/TcpServer.cs @@ -0,0 +1,289 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: TcpServer.cs +* +* TcpServer.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Security; +using System.Threading; +using System.Net.Sockets; +using System.Net.Security; +using System.IO.Pipelines; +using System.Threading.Tasks; +using System.Security.Authentication; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Async; +using VNLib.Utils.Logging; +using VNLib.Utils.Memory.Caching; + +namespace VNLib.Net.Transport.Tcp +{ + /// <summary> + /// <para> + /// Provides a simple, high performance, single process, low/no allocation, + /// asynchronous, TCP socket server. + /// </para> + /// <para> + /// IO operations are full duplex so pipe-lining reused + /// connections is expected. This class cannot be inherited + /// </para> + /// </summary> + public sealed class TcpServer : ICacheHolder + { + /// <summary> + /// The current <see cref="TcpServer"/> configuration + /// </summary> + public TCPConfig Config { get; } + + private readonly ObjectRental<VnSocketAsyncArgs> SockAsyncArgPool; + private readonly PipeOptions PipeOptions; + private readonly bool _usingTls; + + /// <summary> + /// Initializes a new <see cref="TcpServer"/> with the specified <see cref="TCPConfig"/> + /// </summary> + /// <param name="config">Configuration to inalize with</param> + /// <param name="pipeOptions">Optional <see cref="PipeOptions"/> otherwise uses default</param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public TcpServer(TCPConfig config, PipeOptions? pipeOptions = null) + { + Config = config; + //Check config + _ = config.BufferPool ?? throw new ArgumentException("Buffer pool argument cannot be null"); + _ = config.Log ?? throw new ArgumentException("Log argument is required"); + + if (config.MaxRecvBufferData < 4096) + { + throw new ArgumentException("MaxRecvBufferData size must be at least 4096 bytes to avoid data pipeline pefromance issues"); + } + if(config.AcceptThreads < 1) + { + throw new ArgumentException("Accept thread count must be greater than 0"); + } + if(config.AcceptThreads > Environment.ProcessorCount) + { + config.Log.Debug("Suggestion: Setting accept threads to {pc}", Environment.ProcessorCount); + } + //Cache pipe options + PipeOptions = pipeOptions ?? new( + config.BufferPool, + readerScheduler:PipeScheduler.ThreadPool, + writerScheduler:PipeScheduler.ThreadPool, + pauseWriterThreshold: config.MaxRecvBufferData, + minimumSegmentSize: 8192, + useSynchronizationContext:false + ); + //store tls value + _usingTls = Config.AuthenticationOptions != null; + + SockAsyncArgPool = ObjectRental.CreateReusable(ArgsConstructor, Config.CacheQuota); + } + + ///<inheritdoc/> + public void CacheClear() => SockAsyncArgPool.CacheClear(); + ///<inheritdoc/> + public void CacheHardClear() => SockAsyncArgPool.CacheHardClear(); + + private AsyncQueue<VnSocketAsyncArgs>? WaitingSockets; + private Socket? ServerSock; + //private CancellationToken Token; + + private bool _canceledFlag; + + /// <summary> + /// Begins listening for incoming TCP connections on the configured socket + /// </summary> + /// <param name="token">A token that is used to abort listening operations and close the socket</param> + /// <exception cref="SocketException"></exception> + /// <exception cref="SecurityException"></exception> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public void Start(CancellationToken token) + { + //If the socket is still listening + if (ServerSock != null) + { + throw new InvalidOperationException("The server thread is currently listening and cannot be re-started"); + } + //make sure the token isnt already canceled + if (token.IsCancellationRequested) + { + throw new ArgumentException("Token is already canceled", nameof(token)); + } + + //Configure socket on the current thread so exceptions will be raised to the caller + ServerSock = new(Config.LocalEndPoint.AddressFamily, SocketType.Stream, ProtocolType.Tcp); + //Bind socket + ServerSock.Bind(Config.LocalEndPoint); + //Begin listening + ServerSock.Listen(Config.BackLog); + + //See if keepalive should be used + if (Config.TcpKeepalive) + { + //Setup socket keepalive from config + ServerSock.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true); + ServerSock.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval, Config.KeepaliveInterval); + ServerSock.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime, Config.TcpKeepAliveTime); + } + + //Invoke socket created callback + Config.OnSocketCreated?.Invoke(ServerSock); + + //Init waiting socket queue + WaitingSockets = new(false, true); + + //Clear canceled flag + _canceledFlag = false; + + //Start listening for connections + for (int i = 0; i < Config.AcceptThreads; i++) + { + AcceptConnection(); + } + + //Cleanup callback + static void cleanup(object? state) + { + TcpServer server = (TcpServer)state!; + + //Set canceled flag + server._canceledFlag = true; + + //Clean up socket + server.ServerSock!.Dispose(); + server.ServerSock = null; + + server.SockAsyncArgPool.CacheHardClear(); + + //Dispose any queued sockets + while (server.WaitingSockets!.TryDequeue(out VnSocketAsyncArgs? args)) + { + args.Dispose(); + } + } + + //Register cleanup + _ = token.Register(cleanup, this, false); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private VnSocketAsyncArgs ArgsConstructor() + { + void ReturnCb(VnSocketAsyncArgs args) + { + //If the server has exited, dispose the args and dont return to pool + if (_canceledFlag) + { + args.Dispose(); + } + else + { + SockAsyncArgPool.Return(args); + } + } + + //Socket args accept callback functions for this + VnSocketAsyncArgs args = new(AcceptCompleted, ReturnCb, PipeOptions); + return args; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private void AcceptConnection() + { + //Make sure cancellation isnt pending + if (_canceledFlag) + { + return; + } + //Rent new args + VnSocketAsyncArgs acceptArgs = SockAsyncArgPool!.Rent(); + //Accept another socket + if (!acceptArgs.BeginAccept(ServerSock!)) + { + //Completed synchronously + AcceptCompleted(acceptArgs); + } + //Completed async + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private void AcceptCompleted(VnSocketAsyncArgs args) + { + //Examine last op for aborted error, if aborted, then the listening socket has exited + if (args.SocketError == SocketError.OperationAborted) + { + //Dispose args since server is exiting + args.Dispose(); + return; + } + //Check for error on accept, and if no error, enqueue the socket, otherwise disconnect the socket + if (!args.EndAccept() || !WaitingSockets!.TryEnque(args)) + { + //Disconnect the socket (will return the args to the pool) + args.Disconnect(); + } + //Accept a new connection + AcceptConnection(); + } + + + /// <summary> + /// Retreives a connected socket from the waiting queue + /// </summary> + /// <returns>The context of the connect</returns> + /// <exception cref="InvalidOperationException"></exception> + public async ValueTask<TransportEventContext> AcceptAsync(CancellationToken cancellation) + { + _ = WaitingSockets ?? throw new InvalidOperationException("Server is not listening"); + //Args is ready to use + VnSocketAsyncArgs args = await WaitingSockets.DequeueAsync(cancellation); + //See if tls is enabled, if so, start tls handshake + if (_usingTls) + { + //Begin authenication and make sure the socket stream is closed as its required to cleanup + SslStream stream = new(args.Stream, false); + try + { + //auth the new connection + await stream.AuthenticateAsServerAsync(Config.AuthenticationOptions!, cancellation); + return new(args, stream); + } + catch(Exception ex) + { + await stream.DisposeAsync(); + + //Disconnect socket + args.Disconnect(); + + throw new AuthenticationException("Failed client/server TLS authentication", ex); + } + } + else + { + return new(args, args.Stream); + } + } + } +}
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/src/TransportEventContext.cs b/Net.Transport.SimpleTCP/src/TransportEventContext.cs new file mode 100644 index 0000000..fc04d0c --- /dev/null +++ b/Net.Transport.SimpleTCP/src/TransportEventContext.cs @@ -0,0 +1,123 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: TransportEventContext.cs +* +* TransportEventContext.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Net.Sockets; +using System.Net.Security; +using System.Security.Authentication; +using System.Runtime.CompilerServices; +using System.Threading.Tasks; + + +namespace VNLib.Net.Transport.Tcp +{ + /// <summary> + /// Represents the context of a transport connection. It includes the active socket + /// and a stream representing the active transport. + /// </summary> + public readonly struct TransportEventContext + { + /// <summary> + /// The socket referrence to the incoming connection + /// </summary> + private readonly Socket Socket; + + internal readonly VnSocketAsyncArgs _socketArgs; + + /// <summary> + /// The transport security layer security protocol + /// </summary> + public readonly SslProtocols SslVersion { get; } = SslProtocols.None; + /// <summary> + /// A copy of the local endpoint of the listening socket + /// </summary> + public readonly IPEndPoint LocalEndPoint + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => (Socket.LocalEndPoint as IPEndPoint)!; + } + /// <summary> + /// The <see cref="IPEndPoint"/> representing the client's connection information + /// </summary> + public readonly IPEndPoint RemoteEndpoint + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => (Socket.RemoteEndPoint as IPEndPoint)!; + } + /// <summary> + /// The transport stream to be actively read + /// </summary> + public readonly Stream ConnectionStream; + + + internal TransportEventContext(VnSocketAsyncArgs args, Stream @stream) + { + _socketArgs = args; + Socket = args.Socket!; + ConnectionStream = stream; + } + internal TransportEventContext(VnSocketAsyncArgs args, SslStream @stream):this(args, (Stream)stream) + { + SslVersion = stream.SslProtocol; + } + + /// <summary> + /// Closes a connection and cleans up any resources + /// </summary> + /// <param name="ctx"></param> + /// <returns></returns> + public async ValueTask CloseConnectionAsync() + { + //Var to capture ssl shudown exception + Exception? closeExp = null; + + //Verify ssl is being used and the socket is still 'connected' + if (SslVersion > SslProtocols.None && _socketArgs.Socket!.Connected) + { + try + { + await (ConnectionStream as SslStream)!.ShutdownAsync(); + } + catch (Exception ex) + { + closeExp = ex; + } + } + + //dispose the stream and wait for buffered data to be sent + await ConnectionStream.DisposeAsync(); + + //Disconnect + _socketArgs.Disconnect(); + + //if excp occured, re-throw + if (closeExp != null) + { + throw closeExp; + } + } + } +}
\ No newline at end of file diff --git a/Net.Transport.SimpleTCP/src/VNLib.Net.Transport.SimpleTCP.csproj b/Net.Transport.SimpleTCP/src/VNLib.Net.Transport.SimpleTCP.csproj new file mode 100644 index 0000000..7a476da --- /dev/null +++ b/Net.Transport.SimpleTCP/src/VNLib.Net.Transport.SimpleTCP.csproj @@ -0,0 +1,45 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <RootNamespace>VNLib.Net.Transport</RootNamespace> + <Version>1.0.1.4</Version> + <Product>VNLib Simple Transport Library</Product> + <Description>Provides a library for single process asynchronous, event driven, TCP socket listening and supporting structures to implement +simple high performance TCP servers with or without TLS security.</Description> + <Authors>Vaughn Nugent</Authors> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <AssemblyName>VNLib.Net.Transport.SimpleTCP</AssemblyName> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <!-- Resolve nuget dll files and store them in the output dir --> + <PropertyGroup> + <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> + + <Nullable>enable</Nullable> + + <GenerateDocumentationFile>True</GenerateDocumentationFile> + + <AnalysisLevel>latest-all</AnalysisLevel> + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="System.IO.Pipelines" Version="6.0.3" /> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + </ItemGroup> + +</Project> diff --git a/Net.Transport.SimpleTCP/src/VnSocketAsyncArgs.cs b/Net.Transport.SimpleTCP/src/VnSocketAsyncArgs.cs new file mode 100644 index 0000000..9f37762 --- /dev/null +++ b/Net.Transport.SimpleTCP/src/VnSocketAsyncArgs.cs @@ -0,0 +1,181 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Net.Transport.SimpleTCP +* File: VnSocketAsyncArgs.cs +* +* VnSocketAsyncArgs.cs is part of VNLib.Net.Transport.SimpleTCP which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Net.Transport.SimpleTCP is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Net.Transport.SimpleTCP is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net.Sockets; +using System.IO.Pipelines; + +using VNLib.Utils.Memory.Caching; + + +namespace VNLib.Net.Transport.Tcp +{ + internal delegate void SocketCallback(VnSocketAsyncArgs args); + + /// <summary> + /// Reusable <see cref="SocketAsyncEventArgs"/> that manages a pipeline for sending and recieving data. + /// on the connected socket + /// </summary> + internal sealed class VnSocketAsyncArgs : SocketAsyncEventArgs, IReusable + { + private readonly SocketCallback SocketAccepted; + private readonly SocketCallback SocketDisconnected; + + public readonly SocketPipeLineWorker SocketWorker; + + public Socket? Socket => AcceptSocket; + + public VnSocketAsyncArgs(SocketCallback accepted, SocketCallback disconnected, PipeOptions options) : base() + { + SocketWorker = new(options); + SocketAccepted = accepted; + //Only reuse socketes if windows + DisconnectReuseSocket = OperatingSystem.IsWindows(); + SocketDisconnected = disconnected; + } + + public Stream Stream => SocketWorker.NetworkStream; + + /// <summary> + /// Begins an asynchronous accept operation on the current (bound) socket + /// </summary> + /// <param name="sock">The server socket to accept the connection</param> + /// <returns>True if the IO operation is pending</returns> + public bool BeginAccept(Socket sock) + { + //Store the semaphore in the user token event args + SocketError = SocketError.Success; + SocketFlags = SocketFlags.None; + + //Recv during accept is not supported on linux, this flag is set to false on linux + if (DisconnectReuseSocket) + { + //get buffer from the pipe to write initial accept data to + Memory<byte> buffer = SocketWorker.GetMemory(sock.ReceiveBufferSize); + SetBuffer(buffer); + } + + //accept async + return sock.AcceptAsync(this); + } + + /// <summary> + /// Determines if an asynchronous accept operation has completed successsfully + /// and the socket is connected. + /// </summary> + /// <returns>True if the accept was successful, and the accepted socket is connected, false otherwise</returns> + public bool EndAccept() + { + if(SocketError == SocketError.Success) + { + //remove ref to buffer + SetBuffer(null); + //start the socket worker + SocketWorker.Start(Socket!, BytesTransferred); + return true; + } + return false; + } + + /// <summary> + /// Begins an async disconnect operation on a currentl connected socket + /// </summary> + /// <returns>True if the operation is pending</returns> + public void Disconnect() + { + //Clear flags + SocketError = SocketError.Success; + //accept async + if (!Socket!.DisconnectAsync(this)) + { + //Invoke disconnected callback since op completed sync + EndDisconnect(); + //Invoke disconnected callback since op completed sync + SocketDisconnected(this); + } + } + + private void EndDisconnect() + { + //If the disconnection operation failed, do not reuse the socket on next accept + if (SocketError != SocketError.Success) + { + //Dispose the socket before clearing the socket + Socket?.Dispose(); + AcceptSocket = null; + } + } + + protected override void OnCompleted(SocketAsyncEventArgs e) + { + switch (e.LastOperation) + { + case SocketAsyncOperation.Accept: + //Invoke the accepted callback + SocketAccepted(this); + break; + case SocketAsyncOperation.Disconnect: + EndDisconnect(); + //Invoke disconnected callback since op completed sync + SocketDisconnected(this); + break; + default: + throw new InvalidOperationException("Invalid socket operation"); + } + //Clear flags/errors on completion + SocketError = SocketError.Success; + SocketFlags = SocketFlags.None; + } + + void IReusable.Prepare() + { + SocketWorker.Prepare(); + } + + bool IReusable.Release() + { + UserToken = null; + SocketWorker.Release(); + //if the sockeet is connected (or not windows), dispose it and clear the accept socket + if (AcceptSocket?.Connected == true || !DisconnectReuseSocket) + { + AcceptSocket?.Dispose(); + AcceptSocket = null; + } + return true; + } + + public new void Dispose() + { + //Dispose the base class + base.Dispose(); + //Dispose the socket if its set + AcceptSocket?.Dispose(); + AcceptSocket = null; + //Dispose the overlapped stream + SocketWorker.DisposeInternal(); + } + } +} diff --git a/Plugins.Essentials.ServiceStack/LICENSE.txt b/Plugins.Essentials.ServiceStack/LICENSE.txt new file mode 100644 index 0000000..147bcd6 --- /dev/null +++ b/Plugins.Essentials.ServiceStack/LICENSE.txt @@ -0,0 +1,195 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU Affero GPL version 3.0 (or any later version). + +GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/> +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. +Preamble + +The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. + +A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. + +The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. + +An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. + +The precise terms and conditions for copying, distribution and modification follow. +TERMS AND CONDITIONS +0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public License. + +"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based on the Program. + +To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. + +An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. +1. Source Code. + +The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. + +A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same work. +2. Basic Permissions. + +All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. +3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. + +When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. +4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. +5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified it, and giving a relevant date. + b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". + c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. + d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. + +A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. +6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: + + a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. + b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. + c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. + d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. + e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. + +"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. + +If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). + +The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. + +Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. +7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or + b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or + c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or + d) Limiting the use for publicity purposes of names of licensors or authors of the material; or + e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or + f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. + +All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. +8. Termination. + +You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). + +However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. + +Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. + +Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. +9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. +10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. +11. Patents. + +A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. + +If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. + +A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. +12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. +13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. +14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. + +Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. +15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. +16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Plugins.Essentials.ServiceStack/README.md b/Plugins.Essentials.ServiceStack/README.md new file mode 100644 index 0000000..a477635 --- /dev/null +++ b/Plugins.Essentials.ServiceStack/README.md @@ -0,0 +1,65 @@ +# VNLib.Plugins.Essentials.ServiceStack + +This library contains all of the utilities required for an application to build VNLib.Net.Http.HttpServer listeners for nearly unlimited virtual hosts. + +### Breakdown + +**HttpServiceStack** - This immutable data structure represents a collection of listening HttpServer instances across the ServiceDomain, and manages the entire lifecycle of those servers. + +**ServiceDomain** - The immutable collection of ServiceGroups and their dynamically loaded plugins that represent the bindings between the listening servers and their application code. The service domain handles plugin reload events and their dynamic endpoint updates. + +**ServiceGroup** - The immutable collection of service hosts that will share a common HttpServer because they have the same endpoint and TLS/SSL status (enabled or not) + +**IServiceHost** - Represents an immutable container exposing the EventProcessor used to execute host specific operations and the transport information (IHostTransportInfo) + +**IHostTransportInfo** - The service transport information (desired network endpoint to listen on, and an optional X509Certificate) + +**HttpServiceStackBuilder** - The builder class used to build the HttpServiceStack + +## Usage +Again, this library may be used broadly and therefor I will only show basic usage to generate listeners for applications. + +```programming language C# +public static int main (string[] args) +{ + //Start with the new builder + HttpServiceStackBuilder builder = new(); + + //Build the service domain by loading all IServiceHosts + bool built = builder.ServiceDomain.BuildDomain( hostCollection => ... ); + + //Check status + if(!built) + { + return -1; + } + + //Load dynamic plugins + Task loading = builder.ServiceDomain.LoadPlugins(<hostConfig>,<IlogProvider>); + //wait for loading, we don't need to but if plugins don't load we may choose to exit + loading.Wait(); + + //Builds servers by retrieving required ITransportProvider for each service group + builder.BuildServers(<HttpConfig>, group => ... ); + + //Get service stack, in a using statement to cleanup. + using HttpServiceStack serviceStack = builder.ServiceStack; + + //Start servers + serviceStack.StartServers(); + + ... Wait for process exit + + //Stop servers and exit process + serviceStack.StopAndWaitAsync().Wait(); + + return 0; +} +``` + +## License + +The software in this repository is licensed under the GNU Affero General Public License (or any later version). +See the LICENSE files for more information. + + diff --git a/Plugins.Essentials.ServiceStack/src/HttpServiceStack.cs b/Plugins.Essentials.ServiceStack/src/HttpServiceStack.cs new file mode 100644 index 0000000..5800955 --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/HttpServiceStack.cs @@ -0,0 +1,120 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: HttpServiceStack.cs +* +* HttpServiceStack.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using VNLib.Utils; +using VNLib.Net.Http; + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + /// <summary> + /// The service domain controller that manages all + /// servers for an application based on a + /// <see cref="ServiceDomain"/> + /// </summary> + public sealed class HttpServiceStack : VnDisposeable + { + private readonly LinkedList<HttpServer> _servers; + private readonly ServiceDomain _serviceDomain; + + private CancellationTokenSource? _cts; + private Task WaitForAllTask; + + /// <summary> + /// A collection of all loaded servers + /// </summary> + public IReadOnlyCollection<HttpServer> Servers => _servers; + + /// <summary> + /// The service domain's plugin controller + /// </summary> + public IPluginController PluginController => _serviceDomain; + + /// <summary> + /// Initializes a new <see cref="HttpServiceStack"/> that will + /// generate servers to listen for services exposed by the + /// specified host context + /// </summary> + internal HttpServiceStack(LinkedList<HttpServer> servers, ServiceDomain serviceDomain) + { + _servers = servers; + _serviceDomain = serviceDomain; + WaitForAllTask = Task.CompletedTask; + } + + /// <summary> + /// Starts all configured servers that observe a cancellation + /// token to cancel + /// </summary> + /// <param name="parentToken">The token to observe which may stop servers and cleanup the provider</param> + public void StartServers(CancellationToken parentToken = default) + { + Check(); + + //Init new linked cts to stop all servers if cancelled + _cts = CancellationTokenSource.CreateLinkedTokenSource(parentToken); + + LinkedList<Task> runners = new(); + + foreach(HttpServer server in _servers) + { + //Start servers and add run task to list + Task run = server.Start(_cts.Token); + runners.AddLast(run); + } + + //Task that waits for all to exit then cleans up + WaitForAllTask = Task.WhenAll(runners) + .ContinueWith(OnAllServerExit, CancellationToken.None, TaskContinuationOptions.RunContinuationsAsynchronously, TaskScheduler.Default); + } + + /// <summary> + /// Stops listening on all configured servers + /// and returns a task that completes when the service + /// host has stopped all servers and unloaded resources + /// </summary> + /// <returns>The task that completes when</returns> + public Task StopAndWaitAsync() + { + _cts?.Cancel(); + return WaitForAllTask; + } + + private void OnAllServerExit(Task allExit) + { + //Unload the hosts + _serviceDomain.UnloadAll(); + } + + ///<inheritdoc/> + protected override void Free() + { + _cts?.Dispose(); + + _serviceDomain.Dispose(); + + //remove all lists + _servers.Clear(); + } + } +} diff --git a/Plugins.Essentials.ServiceStack/src/HttpServiceStackBuilder.cs b/Plugins.Essentials.ServiceStack/src/HttpServiceStackBuilder.cs new file mode 100644 index 0000000..bb6e96f --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/HttpServiceStackBuilder.cs @@ -0,0 +1,89 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: HttpServiceStackBuilder.cs +* +* HttpServiceStackBuilder.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using VNLib.Net.Http; + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + /// <summary> + /// A data structure used to build/create a <see cref="HttpServiceStack"/> + /// around a <see cref="ServiceDomain"/> + /// </summary> + public sealed class HttpServiceStackBuilder + { + private readonly LinkedList<HttpServer> _servers; + + /// <summary> + /// The built <see cref="HttpServiceStack"/> + /// </summary> + public HttpServiceStack ServiceStack { get; } + + /// <summary> + /// Gets the underlying <see cref="ServiceDomain"/> + /// </summary> + public ServiceDomain ServiceDomain { get; } + + /// <summary> + /// Initializes a new <see cref="HttpServiceStack"/> that will + /// generate servers to listen for services exposed by the + /// specified host context + /// </summary> + public HttpServiceStackBuilder() + { + ServiceDomain = new(); + _servers = new(); + ServiceStack = new(_servers, ServiceDomain); + } + + /// <summary> + /// Builds all http servers from + /// </summary> + /// <param name="config">The http server configuration to user for servers</param> + /// <param name="getTransports">A callback method that gets the transport provider for the given host group</param> + public void BuildServers(in HttpConfig config, Func<ServiceGroup, ITransportProvider> getTransports) + { + //enumerate hosts groups + foreach (ServiceGroup hosts in ServiceDomain.ServiceGroups) + { + //get transport for provider + ITransportProvider transport = getTransports.Invoke(hosts); + + //Create new server + HttpServer server = new(config, transport, hosts.Hosts.Select(static h => h.Processor as IWebRoot)); + + //Add server to internal list + _servers.AddLast(server); + } + } + + /// <summary> + /// Releases any resources that may be held by the <see cref="ServiceDomain"/> + /// incase of an error + /// </summary> + public void ReleaseOnError() + { + ServiceStack.Dispose(); + } + } +} diff --git a/Plugins.Essentials.ServiceStack/src/IHostTransportInfo.cs b/Plugins.Essentials.ServiceStack/src/IHostTransportInfo.cs new file mode 100644 index 0000000..fada93c --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/IHostTransportInfo.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: IHostTransportInfo.cs +* +* IHostTransportInfo.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Security.Cryptography.X509Certificates; +using System.Net; + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + /// <summary> + /// Represents the service host's network/transport + /// information including the optional certificate and + /// the endpoint to listen on + /// </summary> + public interface IHostTransportInfo + { + /// <summary> + /// Optional TLS certificate to use + /// </summary> + X509Certificate? Certificate { get; } + + /// <summary> + /// The endpoint to listen on + /// </summary> + IPEndPoint TransportEndpoint { get; } + } +} diff --git a/Plugins.Essentials.ServiceStack/src/IPluginController.cs b/Plugins.Essentials.ServiceStack/src/IPluginController.cs new file mode 100644 index 0000000..0871fdc --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/IPluginController.cs @@ -0,0 +1,71 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: IPluginController.cs +* +* IPluginController.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Text.Json; + +using VNLib.Utils.Logging; + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + /// <summary> + /// Represents a live plugin controller that manages all + /// plugins loaded in a <see cref="ServiceDomain"/> + /// </summary> + public interface IPluginController + { + /// <summary> + /// Loads all plugins specified by the host config to the service manager, + /// or attempts to load plugins by the default + /// </summary> + /// <param name="config">The configuration instance to pass to plugins</param> + /// <param name="appLog">A log provider to write message and errors to</param> + /// <returns>A task that resolves when all plugins are loaded</returns> + Task LoadPlugins(JsonDocument config, ILogProvider appLog); + + /// <summary> + /// Sends a message to a plugin identified by it's name. + /// </summary> + /// <param name="pluginName">The name of the plugin to pass the message to</param> + /// <param name="message">The message to pass to the plugin</param> + /// <param name="nameComparison">The name string comparison type</param> + /// <returns>True if the plugin was found and it has a message handler loaded</returns> + /// <exception cref="ObjectDisposedException"></exception> + bool SendCommandToPlugin(string pluginName, string message, StringComparison nameComparison = StringComparison.Ordinal); + + /// <summary> + /// Manually reloads all plugins loaded to the current service manager + /// </summary> + /// <exception cref="AggregateException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + void ForceReloadAllPlugins(); + + /// <summary> + /// Unloads all service groups, removes them, and unloads all + /// loaded plugins + /// </summary> + /// <exception cref="AggregateException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + void UnloadAll(); + } +} diff --git a/Plugins.Essentials.ServiceStack/src/IServiceHost.cs b/Plugins.Essentials.ServiceStack/src/IServiceHost.cs new file mode 100644 index 0000000..0c8d6c1 --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/IServiceHost.cs @@ -0,0 +1,42 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: IServiceHost.cs +* +* IServiceHost.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + /// <summary> + /// Represents a host that exposes a processor for host events + /// </summary> + public interface IServiceHost + { + /// <summary> + /// The <see cref="EventProcessor"/> to process + /// incoming HTTP connections + /// </summary> + EventProcessor Processor { get; } + /// <summary> + /// The host's transport infomration + /// </summary> + IHostTransportInfo TransportInfo { get; } + } +} diff --git a/Plugins.Essentials.ServiceStack/src/ServiceDomain.cs b/Plugins.Essentials.ServiceStack/src/ServiceDomain.cs new file mode 100644 index 0000000..c52050a --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/ServiceDomain.cs @@ -0,0 +1,359 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: ServiceDomain.cs +* +* ServiceDomain.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Text.Json; +using System.Diagnostics; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Extensions; +using VNLib.Utils.Logging; +using VNLib.Plugins.Runtime; +using VNLib.Plugins.Essentials.Content; +using VNLib.Plugins.Essentials.Sessions; + + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + /// <summary> + /// Represents a domain of services and thier dynamically loaded plugins + /// that will be hosted by an application service stack + /// </summary> + public sealed class ServiceDomain : VnDisposeable, IPluginController + { + private const string PLUGIN_FILE_EXTENSION = ".dll"; + private const string DEFUALT_PLUGIN_DIR = "/plugins"; + private const string PLUGINS_CONFIG_ELEMENT = "plugins"; + + private readonly LinkedList<ServiceGroup> _serviceGroups; + private readonly LinkedList<RuntimePluginLoader> _pluginLoaders; + + /// <summary> + /// Enumerates all loaded plugin instances + /// </summary> + public IEnumerable<IPlugin> Plugins => _pluginLoaders.SelectMany(static s => + s.LivePlugins.Where(static p => p.Plugin != null) + .Select(static s => s.Plugin!) + ); + + /// <summary> + /// Gets all service groups loaded in the service manager + /// </summary> + public IReadOnlyCollection<ServiceGroup> ServiceGroups => _serviceGroups; + + /// <summary> + /// Initializes a new empty <see cref="ServiceDomain"/> + /// </summary> + public ServiceDomain() + { + _serviceGroups = new(); + _pluginLoaders = new(); + } + + /// <summary> + /// Uses the supplied callback to get a collection of virtual hosts + /// to build the current domain with + /// </summary> + /// <param name="hostBuilder">The callback method to build virtual hosts</param> + /// <returns>A value that indicates if any virtual hosts were successfully loaded</returns> + public bool BuildDomain(Action<ICollection<IServiceHost>> hostBuilder) + { + //LL to store created hosts + LinkedList<IServiceHost> hosts = new(); + + //build hosts + hostBuilder.Invoke(hosts); + + return FromExisting(hosts); + } + + /// <summary> + /// Builds the domain from an existing enumeration of virtual hosts + /// </summary> + /// <param name="hosts">The enumeration of virtual hosts</param> + /// <returns>A value that indicates if any virtual hosts were successfully loaded</returns> + public bool FromExisting(IEnumerable<IServiceHost> hosts) + { + //Get service groups and pass service group list + CreateServiceGroups(_serviceGroups, hosts); + return _serviceGroups.Any(); + } + + private static void CreateServiceGroups(ICollection<ServiceGroup> groups, IEnumerable<IServiceHost> hosts) + { + //Get distinct interfaces + IEnumerable<IPEndPoint> interfaces = hosts.Select(static s => s.TransportInfo.TransportEndpoint).Distinct(); + + //Select hosts of the same interface to create a group from + foreach (IPEndPoint iface in interfaces) + { + IEnumerable<IServiceHost> groupHosts = hosts.Where(host => host.TransportInfo.TransportEndpoint.Equals(iface)); + + IServiceHost[]? overlap = groupHosts.Where(vh => groupHosts.Select(static s => s.Processor.Hostname).Count(hostname => vh.Processor.Hostname == hostname) > 1).ToArray(); + + foreach (IServiceHost vh in overlap) + { + throw new ArgumentException($"The hostname '{vh.Processor.Hostname}' is already in use by another virtual host"); + } + + //init new service group around an interface and its roots + ServiceGroup group = new(iface, groupHosts); + + groups.Add(group); + } + } + + ///<inheritdoc/> + public Task LoadPlugins(JsonDocument config, ILogProvider appLog) + { + if (!config.RootElement.TryGetProperty(PLUGINS_CONFIG_ELEMENT, out JsonElement pluginEl)) + { + appLog.Information("Plugins element not defined in config, skipping plugin loading"); + return Task.CompletedTask; + } + + //Get the plugin directory, or set to default + string pluginDir = pluginEl.GetPropString("path") ?? Path.Combine(Directory.GetCurrentDirectory(), DEFUALT_PLUGIN_DIR); + //Get the hot reload flag + bool hotReload = pluginEl.TryGetProperty("hot_reload", out JsonElement hrel) && hrel.GetBoolean(); + + //Load all virtual file assemblies withing the plugin folder + DirectoryInfo dir = new(pluginDir); + + if (!dir.Exists) + { + appLog.Warn("Plugin directory {dir} does not exist. No plugins were loaded", pluginDir); + return Task.CompletedTask; + } + + appLog.Information("Loading plugins. Hot-reload: {en}", hotReload); + + //Enumerate all dll files within this dir + IEnumerable<DirectoryInfo> dirs = dir.EnumerateDirectories("*", SearchOption.TopDirectoryOnly); + + //Select only dirs with a dll that is named after the directory name + IEnumerable<string> pluginPaths = dirs.Where(static pdir => + { + string compined = Path.Combine(pdir.FullName, pdir.Name); + string FilePath = string.Concat(compined, PLUGIN_FILE_EXTENSION); + return FileOperations.FileExists(FilePath); + }) + //Return the name of the dll file to import + .Select(static pdir => + { + string compined = Path.Combine(pdir.FullName, pdir.Name); + return string.Concat(compined, PLUGIN_FILE_EXTENSION); + }); + + IEnumerable<string> pluginFileNames = pluginPaths.Select(static s => $"{Path.GetFileName(s)}\n"); + + appLog.Debug("Found plugin files: \n{files}", string.Concat(pluginFileNames)); + + LinkedList<Task> loading = new(); + + object listLock = new(); + + foreach (string pluginPath in pluginPaths) + { + async Task Load() + { + string pluginName = Path.GetFileName(pluginPath); + + RuntimePluginLoader plugin = new(pluginPath, config, appLog, hotReload, hotReload); + Stopwatch sw = new(); + try + { + sw.Start(); + await plugin.InitLoaderAsync(); + //Listen for reload events to remove and re-add endpoints + plugin.Reloaded += OnPluginReloaded; + + lock (listLock) + { + //Add to list + _pluginLoaders.AddLast(plugin); + } + + sw.Stop(); + + appLog.Verbose("Loaded {pl} in {tm} ms", pluginName, sw.ElapsedMilliseconds); + } + catch (Exception ex) + { + appLog.Error(ex, $"Exception raised during loading {pluginName}. Failed to load plugin \n{ex}"); + plugin.Dispose(); + } + finally + { + sw.Stop(); + } + } + + loading.AddLast(Load()); + } + + //Continuation to add all initial plugins to the service manager + void Continuation(Task t) + { + appLog.Verbose("Plugins loaded"); + + //Add inital endpoints for all plugins + _pluginLoaders.TryForeach(ldr => _serviceGroups.TryForeach(sg => sg.AddOrUpdateEndpointsForPlugin(ldr))); + + //Init session provider + InitSessionProvider(); + + //Init page router + InitPageRouter(); + } + + //wait for loading to completed + return Task.WhenAll(loading.ToArray()).ContinueWith(Continuation, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + } + + ///<inheritdoc/> + public bool SendCommandToPlugin(string pluginName, string message, StringComparison nameComparison = StringComparison.Ordinal) + { + Check(); + //Find the single plugin by its name + LivePlugin? pl = _pluginLoaders.Select(p => + p.LivePlugins.Where(lp => pluginName.Equals(lp.PluginName, nameComparison)) + ) + .SelectMany(static lp => lp) + .SingleOrDefault(); + //Send the command + return pl?.SendConsoleMessage(message) ?? false; + } + + ///<inheritdoc/> + public void ForceReloadAllPlugins() + { + Check(); + _pluginLoaders.TryForeach(static pl => pl.ReloadPlugin()); + } + + ///<inheritdoc/> + public void UnloadAll() + { + Check(); + + //Unload service groups before unloading plugins + _serviceGroups.TryForeach(static sg => sg.UnloadAll()); + //empty service groups + _serviceGroups.Clear(); + + //Unload all plugins + _pluginLoaders.TryForeach(static pl => pl.UnloadAll()); + } + + private void OnPluginReloaded(object? plugin, EventArgs empty) + { + //Update endpoints for the loader + RuntimePluginLoader reloaded = (plugin as RuntimePluginLoader)!; + + //Update all endpoints for the plugin + _serviceGroups.TryForeach(sg => sg.AddOrUpdateEndpointsForPlugin(reloaded)); + } + + private void InitSessionProvider() + { + //Callback to reload provider + void onSessionProviderReloaded(ISessionProvider old, ISessionProvider current) + { + _serviceGroups.TryForeach(sg => sg.UpdateSessionProvider(current)); + } + + try + { + //get the loader that contains the single session provider + RuntimePluginLoader? sessionLoader = _pluginLoaders + .Where(static s => s.ExposesType<ISessionProvider>()) + .SingleOrDefault(); + + //If session provider has been supplied, load it + if (sessionLoader != null) + { + //Get the session provider from the plugin loader + ISessionProvider sp = sessionLoader.GetExposedTypeFromPlugin<ISessionProvider>()!; + + //Init inital provider + onSessionProviderReloaded(null!, sp); + + //Register reload event + sessionLoader.RegisterListenerForSingle<ISessionProvider>(onSessionProviderReloaded); + } + } + catch (InvalidOperationException) + { + throw new TypeLoadException("More than one session provider plugin was defined in the plugin directory, cannot continue"); + } + } + + private void InitPageRouter() + { + //Callback to reload provider + void onRouterReloaded(IPageRouter old, IPageRouter current) + { + _serviceGroups.TryForeach(sg => sg.UpdatePageRouter(current)); + } + + try + { + + //get the loader that contains the single page router + RuntimePluginLoader? routerLoader = _pluginLoaders + .Where(static s => s.ExposesType<IPageRouter>()) + .SingleOrDefault(); + + //If router has been supplied, load it + if (routerLoader != null) + { + //Get initial value + IPageRouter sp = routerLoader.GetExposedTypeFromPlugin<IPageRouter>()!; + + //Init inital provider + onRouterReloaded(null!, sp); + + //Register reload event + routerLoader.RegisterListenerForSingle<IPageRouter>(onRouterReloaded); + } + } + catch (InvalidOperationException) + { + throw new TypeLoadException("More than one page router plugin was defined in the plugin directory, cannot continue"); + } + } + + ///<inheritdoc/> + protected override void Free() + { + //Dispose loaders + _pluginLoaders.TryForeach(static pl => pl.Dispose()); + _pluginLoaders.Clear(); + _serviceGroups.Clear(); + } + } +} diff --git a/Plugins.Essentials.ServiceStack/src/ServiceGroup.cs b/Plugins.Essentials.ServiceStack/src/ServiceGroup.cs new file mode 100644 index 0000000..f57a6f9 --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/ServiceGroup.cs @@ -0,0 +1,128 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials.ServiceStack +* File: ServiceGroup.cs +* +* ServiceGroup.cs is part of VNLib.Plugins.Essentials.ServiceStack which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials.ServiceStack is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 2 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials.ServiceStack is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Net; +using System.Collections.Generic; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Extensions; +using VNLib.Plugins.Runtime; +using VNLib.Plugins.Essentials.Content; +using VNLib.Plugins.Essentials.Sessions; + +namespace VNLib.Plugins.Essentials.ServiceStack +{ + + /// <summary> + /// Represents a collection of virtual hosts that share a + /// common transport (interface, port, and SSL status) + /// and may be loaded by a single server instance. + /// </summary> + public sealed class ServiceGroup + { + private readonly LinkedList<IServiceHost> _vHosts; + private readonly ConditionalWeakTable<RuntimePluginLoader, IEndpoint[]> _endpointsForPlugins; + + /// <summary> + /// The <see cref="IPEndPoint"/> transport endpoint for all loaded service hosts + /// </summary> + public IPEndPoint ServiceEndpoint { get; } + + /// <summary> + /// The collection of hosts that are loaded by this group + /// </summary> + public IReadOnlyCollection<IServiceHost> Hosts => _vHosts; + + /// <summary> + /// Initalizes a new <see cref="ServiceGroup"/> of virtual hosts + /// with common transport + /// </summary> + /// <param name="serviceEndpoint">The <see cref="IPEndPoint"/> to listen for connections on</param> + /// <param name="hosts">The hosts that share a common interface endpoint</param> + public ServiceGroup(IPEndPoint serviceEndpoint, IEnumerable<IServiceHost> hosts) + { + _endpointsForPlugins = new(); + _vHosts = new(hosts); + ServiceEndpoint = serviceEndpoint; + } + + /// <summary> + /// Sets the specified page rotuer for all virtual hosts + /// </summary> + /// <param name="router">The page router to user</param> + internal void UpdatePageRouter(IPageRouter router) => _vHosts.TryForeach(v => v.Processor.SetPageRouter(router)); + /// <summary> + /// Sets the specified session provider for all virtual hosts + /// </summary> + /// <param name="current">The session provider to use</param> + internal void UpdateSessionProvider(ISessionProvider current) => _vHosts.TryForeach(v => v.Processor.SetSessionProvider(current)); + + /// <summary> + /// Adds or updates all endpoints exported by all plugins + /// within the specified loader. All endpoints exposed + /// by a previously loaded instance are removed and all + /// currently exposed endpoints are added to all virtual + /// hosts + /// </summary> + /// <param name="loader">The plugin loader to get add/update endpoints from</param> + internal void AddOrUpdateEndpointsForPlugin(RuntimePluginLoader loader) + { + //Get all new endpoints for plugin + IEndpoint[] newEndpoints = loader.LivePlugins.SelectMany(static pl => pl.Plugin!.GetEndpoints()).ToArray(); + + //See if + if(_endpointsForPlugins.TryGetValue(loader, out IEndpoint[]? oldEps)) + { + //Remove old endpoints + _vHosts.TryForeach(v => v.Processor.RemoveEndpoint(oldEps)); + } + + //Add endpoints to dict + _endpointsForPlugins.AddOrUpdate(loader, newEndpoints); + + //Add endpoints to hosts + _vHosts.TryForeach(v => v.Processor.AddEndpoint(newEndpoints)); + } + + /// <summary> + /// Unloads all previously stored endpoints, router, session provider, and + /// clears all internal data structures + /// </summary> + internal void UnloadAll() + { + //Remove all loaded endpoints + _vHosts.TryForeach(v => _endpointsForPlugins.TryForeach(eps => v.Processor.RemoveEndpoint(eps.Value))); + + //Remove all routers + _vHosts.TryForeach(static v => v.Processor.SetPageRouter(null)); + //Remove all session providers + _vHosts.TryForeach(static v => v.Processor.SetSessionProvider(null)); + + //Clear all hosts + _vHosts.Clear(); + //Clear all endpoints + _endpointsForPlugins.Clear(); + } + } +} diff --git a/Plugins.Essentials.ServiceStack/src/VNLib.Plugins.Essentials.ServiceStack.csproj b/Plugins.Essentials.ServiceStack/src/VNLib.Plugins.Essentials.ServiceStack.csproj new file mode 100644 index 0000000..a604ce9 --- /dev/null +++ b/Plugins.Essentials.ServiceStack/src/VNLib.Plugins.Essentials.ServiceStack.csproj @@ -0,0 +1,40 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <ImplicitUsings>enable</ImplicitUsings> + <Nullable>enable</Nullable> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <Authors>Vaughn Nugent</Authors> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <Version>1.0.1.2</Version> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <PackageReadmeFile>README.md</PackageReadmeFile> + <AnalysisLevel>latest-all</AnalysisLevel> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <ItemGroup> + <None Include="..\README.md"> + <Pack>True</Pack> + <PackagePath>\</PackagePath> + </None> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Essentials\src\VNLib.Plugins.Essentials.csproj" /> + <ProjectReference Include="..\..\Http\src\VNLib.Net.Http.csproj" /> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + <ProjectReference Include="..\..\VNLib.Plugins.Runtime\src\VNLib.Plugins.Runtime.csproj" /> + </ItemGroup> + +</Project> diff --git a/Plugins.Essentials/LICENSE.txt b/Plugins.Essentials/LICENSE.txt new file mode 100644 index 0000000..147bcd6 --- /dev/null +++ b/Plugins.Essentials/LICENSE.txt @@ -0,0 +1,195 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU Affero GPL version 3.0 (or any later version). + +GNU AFFERO GENERAL PUBLIC LICENSE + +Version 3, 19 November 2007 + +Copyright © 2007 Free Software Foundation, Inc. <https://fsf.org/> +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. +Preamble + +The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. + +The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. + +When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. + +Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. + +A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. + +The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. + +An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. + +The precise terms and conditions for copying, distribution and modification follow. +TERMS AND CONDITIONS +0. Definitions. + +"This License" refers to version 3 of the GNU Affero General Public License. + +"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based on the Program. + +To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. + +An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. +1. Source Code. + +The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. + +A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same work. +2. Basic Permissions. + +All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. +3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. + +When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. +4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. +5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified it, and giving a relevant date. + b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". + c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. + d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. + +A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. +6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: + + a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. + b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. + c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. + d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. + e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. + +"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. + +If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). + +The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. + +Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. +7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or + b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or + c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or + d) Limiting the use for publicity purposes of names of licensors or authors of the material; or + e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or + f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. + +All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. +8. Termination. + +You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). + +However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. + +Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. + +Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. +9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. +10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. +11. Patents. + +A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. + +If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. + +A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. +12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. +13. Remote Network Interaction; Use with the GNU General Public License. + +Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. + +Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. +14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. + +If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. + +Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. +15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. +16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. +17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Plugins.Essentials/README.md b/Plugins.Essentials/README.md new file mode 100644 index 0000000..2399b61 --- /dev/null +++ b/Plugins.Essentials/README.md @@ -0,0 +1,3 @@ +# VNLib.Plugins.Essentials + +A library to add essential http processing features to your VNLib http application. This library is a part of the VNLib project and mostly focuses on the http content processing. It provides a layer between the base HTTP protocol and serving web content. It provides essential processing layers, extension methods, helper namespaces and more. Some features include stateful sessions, dynamic content routing, runtime extensible plugins, and abstractions for building plugins.
\ No newline at end of file diff --git a/Plugins.Essentials/src/Accounts/AccountData.cs b/Plugins.Essentials/src/Accounts/AccountData.cs new file mode 100644 index 0000000..d4a4d12 --- /dev/null +++ b/Plugins.Essentials/src/Accounts/AccountData.cs @@ -0,0 +1,52 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: AccountData.cs +* +* AccountData.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Text.Json.Serialization; + +namespace VNLib.Plugins.Essentials.Accounts +{ + public class AccountData + { + [JsonPropertyName("email")] + public string EmailAddress { get; set; } + [JsonPropertyName("phone")] + public string PhoneNumber { get; set; } + [JsonPropertyName("first")] + public string First { get; set; } + [JsonPropertyName("last")] + public string Last { get; set; } + [JsonPropertyName("company")] + public string Company { get; set; } + [JsonPropertyName("street")] + public string Street { get; set; } + [JsonPropertyName("city")] + public string City { get; set; } + [JsonPropertyName("state")] + public string State { get; set; } + [JsonPropertyName("zip")] + public string Zip { get; set; } + [JsonPropertyName("created")] + public string Created { get; set; } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Accounts/AccountManager.cs b/Plugins.Essentials/src/Accounts/AccountManager.cs new file mode 100644 index 0000000..f148fdb --- /dev/null +++ b/Plugins.Essentials/src/Accounts/AccountManager.cs @@ -0,0 +1,872 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: AccountManager.cs +* +* AccountManager.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Threading.Tasks; +using System.Security.Cryptography; +using System.Text.RegularExpressions; +using System.Runtime.CompilerServices; + +using VNLib.Hashing; +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; +using VNLib.Plugins.Essentials.Users; +using VNLib.Plugins.Essentials.Sessions; +using VNLib.Plugins.Essentials.Extensions; + + +#nullable enable + +namespace VNLib.Plugins.Essentials.Accounts +{ + + /// <summary> + /// Provides essential constants, static methods, and session/user extensions + /// to facilitate unified user-controls, athentication, and security + /// application-wide + /// </summary> + public static partial class AccountManager + { + public const int MAX_EMAIL_CHARS = 50; + public const int ID_FIELD_CHARS = 65; + public const int STREET_ADDR_CHARS = 150; + public const int MAX_LOGIN_COUNT = 10; + public const int MAX_FAILED_RESET_ATTEMPS = 5; + + /// <summary> + /// The maximum time in seconds for a login message to be considered valid + /// </summary> + public const double MAX_TIME_DIFF_SECS = 10.00; + /// <summary> + /// The size in bytes of the random passwords generated when invoking the <see cref="SetRandomPasswordAsync(PasswordHashing, IUserManager, IUser, int)"/> + /// </summary> + public const int RANDOM_PASS_SIZE = 128; + /// <summary> + /// The name of the header that will identify a client's identiy + /// </summary> + public const string LOGIN_TOKEN_HEADER = "X-Web-Token"; + /// <summary> + /// The origin string of a local user account. This value will be set if an + /// account is created through the VNLib.Plugins.Essentials.Accounts library + /// </summary> + public const string LOCAL_ACCOUNT_ORIGIN = "local"; + /// <summary> + /// The size (in bytes) of the challenge secret + /// </summary> + public const int CHALLENGE_SIZE = 64; + /// <summary> + /// The size (in bytes) of the sesssion long user-password challenge + /// </summary> + public const int SESSION_CHALLENGE_SIZE = 128; + + //The buffer size to use when decoding the base64 public key from the user + private const int PUBLIC_KEY_BUFFER_SIZE = 1024; + /// <summary> + /// The name of the login cookie set when a user logs in + /// </summary> + public const string LOGIN_COOKIE_NAME = "VNLogin"; + /// <summary> + /// The name of the login client identifier cookie (cookie that is set fir client to use to determine if the user is logged in) + /// </summary> + public const string LOGIN_COOKIE_IDENTIFIER = "li"; + + private const int LOGIN_COOKIE_SIZE = 64; + + //Session entry keys + private const string BROWSER_ID_ENTRY = "acnt.bid"; + private const string CLIENT_PUB_KEY_ENTRY = "acnt.pbk"; + private const string CHALLENGE_HMAC_ENTRY = "acnt.cdig"; + private const string FAILED_LOGIN_ENTRY = "acnt.flc"; + private const string LOCAL_ACCOUNT_ENTRY = "acnt.ila"; + private const string ACC_ORIGIN_ENTRY = "__.org"; + //private const string CHALLENGE_HASH_ENTRY = "acnt.chl"; + + //Privlage masks + public const ulong READ_MSK = 0x0000000000000001L; + public const ulong DOWNLOAD_MSK = 0x0000000000000002L; + public const ulong WRITE_MSK = 0x0000000000000004L; + public const ulong DELETE_MSK = 0x0000000000000008L; + public const ulong ALLFILE_MSK = 0x000000000000000FL; + public const ulong OPTIONS_MSK = 0x000000000000FF00L; + public const ulong GROUP_MSK = 0x00000000FFFF0000L; + public const ulong LEVEL_MSK = 0x000000FF00000000L; + + public const byte OPTIONS_MSK_OFFSET = 0x08; + public const byte GROUP_MSK_OFFSET = 0x10; + public const byte LEVEL_MSK_OFFSET = 0x18; + + public const ulong MINIMUM_LEVEL = 0x0000000100000001L; + + //Timeouts + public static readonly TimeSpan LoginCookieLifespan = TimeSpan.FromHours(1); + public static readonly TimeSpan RegenIdPeriod = TimeSpan.FromMinutes(25); + + /// <summary> + /// The client data encryption padding. + /// </summary> + public static readonly RSAEncryptionPadding ClientEncryptonPadding = RSAEncryptionPadding.OaepSHA256; + + /// <summary> + /// The size (in bytes) of the web-token hash size + /// </summary> + private static readonly int TokenHashSize = (SHA384.Create().HashSize / 8); + + /// <summary> + /// Speical character regual expresion for basic checks + /// </summary> + public static readonly Regex SpecialCharacters = new(@"[\r\n\t\a\b\e\f#?!@$%^&*\+\-\~`|<>\{}]", RegexOptions.Compiled); + + #region Password/User helper extensions + + /// <summary> + /// Generates and sets a random password for the specified user account + /// </summary> + /// <param name="manager">The configured <see cref="IUserManager"/> to process the password update on</param> + /// <param name="user">The user instance to update the password on</param> + /// <param name="passHashing">The <see cref="PasswordHashing"/> instance to hash the random password with</param> + /// <param name="size">Size (in bytes) of the generated random password</param> + /// <returns>A value indicating the results of the event (number of rows affected, should evaluate to true)</returns> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + public static async Task<ERRNO> SetRandomPasswordAsync(this PasswordHashing passHashing, IUserManager manager, IUser user, int size = RANDOM_PASS_SIZE) + { + _ = manager ?? throw new ArgumentNullException(nameof(manager)); + _ = user ?? throw new ArgumentNullException(nameof(user)); + _ = passHashing ?? throw new ArgumentNullException(nameof(passHashing)); + if (user.IsReleased) + { + throw new ObjectDisposedException("The specifed user object has been released"); + } + //Alloc a buffer + using IMemoryHandle<byte> buffer = Memory.SafeAlloc<byte>(size); + //Use the CGN to get a random set + RandomHash.GetRandomBytes(buffer.Span); + //Hash the new random password + using PrivateString passHash = passHashing.Hash(buffer.Span); + //Write the password to the user account + return await manager.UpdatePassAsync(user, passHash); + } + + + /// <summary> + /// Checks to see if the current user account was created + /// using a local account. + /// </summary> + /// <param name="user"></param> + /// <returns>True if the account is a local account, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsLocalAccount(this IUser user) => LOCAL_ACCOUNT_ORIGIN.Equals(user.GetAccountOrigin(), StringComparison.Ordinal); + + /// <summary> + /// If this account was created by any means other than a local account creation. + /// Implementors can use this method to determine the origin of the account. + /// This field is not required + /// </summary> + /// <returns>The origin of the account</returns> + public static string GetAccountOrigin(this IUser ud) => ud[ACC_ORIGIN_ENTRY]; + /// <summary> + /// If this account was created by any means other than a local account creation. + /// Implementors can use this method to specify the origin of the account. This field is not required + /// </summary> + /// <param name="ud"></param> + /// <param name="origin">Value of the account origin</param> + public static void SetAccountOrigin(this IUser ud, string origin) => ud[ACC_ORIGIN_ENTRY] = origin; + + /// <summary> + /// Gets a random user-id generated from crypograhic random number + /// then hashed (SHA1) and returns a hexadecimal string + /// </summary> + /// <returns>The random string user-id</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetRandomUserId() => RandomHash.GetRandomHash(HashAlg.SHA1, 64, HashEncodingMode.Hexadecimal); + + #endregion + + #region Client Auth Extensions + + /// <summary> + /// Runs necessary operations to grant authorization to the specified user of a given session and user with provided variables + /// </summary> + /// <param name="ev">The connection and session to log-in</param> + /// <param name="loginMessage">The message of the client to set the log-in status of</param> + /// <param name="user">The user to log-in</param> + /// <returns>The encrypted base64 token secret data to send to the client</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="CryptographicException"></exception> + public static string GenerateAuthorization(this HttpEntity ev, LoginMessage loginMessage, IUser user) + { + return GenerateAuthorization(ev, loginMessage.ClientPublicKey, loginMessage.ClientID, user); + } + + /// <summary> + /// Runs necessary operations to grant authorization to the specified user of a given session and user with provided variables + /// </summary> + /// <param name="ev">The connection and session to log-in</param> + /// <param name="base64PubKey">The clients base64 public key</param> + /// <param name="clientId">The browser/client id</param> + /// <param name="user">The user to log-in</param> + /// <returns>The encrypted base64 token secret data to send to the client</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="CryptographicException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public static string GenerateAuthorization(this HttpEntity ev, string base64PubKey, string clientId, IUser user) + { + if (!ev.Session.IsSet || ev.Session.SessionType != SessionType.Web) + { + throw new InvalidOperationException("The session is not set or the session is not a web-based session type"); + } + //derrive token from login data + TryGenerateToken(base64PubKey, out string base64ServerToken, out string base64ClientData); + //Clear flags + user.FailedLoginCount(0); + //Get the "local" account flag from the user object + bool localAccount = user.IsLocalAccount(); + //Set login cookie and session login hash + ev.SetLogin(localAccount); + //Store variables + ev.Session.UserID = user.UserID; + ev.Session.Privilages = user.Privilages; + //Store browserid/client id if specified + SetBrowserID(in ev.Session, clientId); + //Store the clients public key + SetBrowserPubKey(in ev.Session, base64PubKey); + //Set local account flag + ev.Session.HasLocalAccount(localAccount); + //Store the base64 server key to compute the hmac later + ev.Session.Token = base64ServerToken; + //Return the client encrypted data + return base64ClientData; + } + + /* + * Notes for RSA client token generator code below + * + * To log-in a client with the following API the calling code + * must have already determined that the client should be + * logged in (verified passwords or auth tokens). + * + * The client will send a LoginMessage object that will + * contain the following Information. + * + * - The clients RSA public key in base64 subject-key info format + * - The client browser's id hex string + * - The clients local-time + * + * The TryGenerateToken method, will generate a random-byte token, + * encrypt it using the clients RSA public key, return the encrypted + * token data to the client, and only the client will be able to + * decrypt the token data. + * + * The token data is also hashed with SHA-256 (for future use) and + * stored in the client's session store. The client must decrypt + * the token data, hash it, and return it as a header for verification. + * + * Ideally the client should sign the data and send the signature or + * hash back, but it wont prevent MITM, and for now I think it just + * adds extra overhead for every connection during the HttpEvent.TokenMatches() + * check extension method + */ + + private ref struct TokenGenBuffers + { + public readonly Span<byte> Buffer { private get; init; } + public readonly Span<byte> SignatureBuffer => Buffer[..64]; + + + + public int ClientPbkWritten; + public readonly Span<byte> ClientPublicKeyBuffer => Buffer.Slice(64, 1024); + public readonly ReadOnlySpan<byte> ClientPbkOutput => ClientPublicKeyBuffer[..ClientPbkWritten]; + + + + public int ClientEncBytesWritten; + public readonly Span<byte> ClientEncOutputBuffer => Buffer[(64 + 1024)..]; + public readonly ReadOnlySpan<byte> EncryptedOutput => ClientEncOutputBuffer[..ClientEncBytesWritten]; + } + + /// <summary> + /// Computes a random buffer, encrypts it with the client's public key, + /// computes the digest of that key and returns the base64 encoded strings + /// of those components + /// </summary> + /// <param name="base64clientPublicKey">The user's public key credential</param> + /// <param name="base64Digest">The base64 encoded digest of the secret that was encrypted</param> + /// <param name="base64ClientData">The client's user-agent header value</param> + /// <returns>A string representing a unique signed token for a given login context</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="CryptographicException"></exception> + private static void TryGenerateToken(string base64clientPublicKey, out string base64Digest, out string base64ClientData) + { + //Temporary work buffer + using IMemoryHandle<byte> buffer = Memory.SafeAlloc<byte>(4096, true); + /* + * Create a new token buffer for bin buffers. + * This buffer struct is used to break up + * a single block of memory into individual + * non-overlapping (important!) buffer windows + * for named purposes + */ + TokenGenBuffers tokenBuf = new() + { + Buffer = buffer.Span + }; + //Recover the clients public key from its base64 encoding + if (!Convert.TryFromBase64String(base64clientPublicKey, tokenBuf.ClientPublicKeyBuffer, out tokenBuf.ClientPbkWritten)) + { + throw new InternalBufferOverflowException("Failed to recover the clients RSA public key"); + } + /* + * Fill signature buffer with random data + * this signature will be stored and used to verify + * signed client messages. It will also be encryped + * using the clients RSA keys + */ + RandomHash.GetRandomBytes(tokenBuf.SignatureBuffer); + /* + * Setup a new RSA Crypto provider that is initialized with the clients + * supplied public key. RSA will be used to encrypt the server secret + * that only the client will be able to decrypt for the current connection + */ + using RSA rsa = RSA.Create(); + //Setup rsa from the users public key + rsa.ImportSubjectPublicKeyInfo(tokenBuf.ClientPbkOutput, out _); + //try to encypte output data + if (!rsa.TryEncrypt(tokenBuf.SignatureBuffer, tokenBuf.ClientEncOutputBuffer, RSAEncryptionPadding.OaepSHA256, out tokenBuf.ClientEncBytesWritten)) + { + throw new InternalBufferOverflowException("Failed to encrypt the server secret"); + } + //Compute the digest of the raw server key + base64Digest = ManagedHash.ComputeBase64Hash(tokenBuf.SignatureBuffer, HashAlg.SHA384); + /* + * The client will send a hash of the decrypted key and will be used + * as a comparison to the hash string above ^ + */ + base64ClientData = Convert.ToBase64String(tokenBuf.EncryptedOutput, Base64FormattingOptions.None); + } + + /// <summary> + /// Determines if the client sent a token header, and it maches against the current session + /// </summary> + /// <returns>true if the client set the token header, the session is loaded, and the token matches the session, false otherwise</returns> + public static bool TokenMatches(this HttpEntity ev) + { + //Get the token from the client header, the client should always sent this + string? clientDigest = ev.Server.Headers[LOGIN_TOKEN_HEADER]; + //Make sure a session is loaded + if (!ev.Session.IsSet || ev.Session.IsNew || string.IsNullOrWhiteSpace(clientDigest)) + { + return false; + } + /* + * Alloc buffer to do conversion and zero initial contents incase the + * payload size has been changed. + * + * The buffer just needs to be large enoguh for the size of the hashes + * that are stored in base64 format. + * + * The values in the buffers will be the raw hash of the client's key + * and the stored key sent during initial authorziation. If the hashes + * are equal it should mean that the client must have the private + * key that generated the public key that was sent + */ + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(TokenHashSize * 2, true); + //Slice up buffers + Span<byte> headerBuffer = buffer.Span[..TokenHashSize]; + Span<byte> sessionBuffer = buffer.Span[TokenHashSize..]; + //Convert the header token and the session token + if (Convert.TryFromBase64String(clientDigest, headerBuffer, out int headerTokenLen) + && Convert.TryFromBase64String(ev.Session.Token, sessionBuffer, out int sessionTokenLen)) + { + //Do a fixed time equal (probably overkill, but should not matter too much) + return CryptographicOperations.FixedTimeEquals(headerBuffer[..headerTokenLen], sessionBuffer[..sessionTokenLen]); + } + return false; + } + + /// <summary> + /// Regenerates the user's login token with the public key stored + /// during initial logon + /// </summary> + /// <returns>The base64 of the newly encrypted secret</returns> + public static string? RegenerateClientToken(this HttpEntity ev) + { + if(!ev.Session.IsSet || ev.Session.SessionType != SessionType.Web) + { + return null; + } + //Get the client's stored public key + string clientPublicKey = ev.Session.GetBrowserPubKey(); + //Make sure its set + if (string.IsNullOrWhiteSpace(clientPublicKey)) + { + return null; + } + //Generate a new token using the stored public key + TryGenerateToken(clientPublicKey, out string base64Digest, out string base64ClientData); + //store the token to the user's session + ev.Session.Token = base64Digest; + //return the clients encrypted secret + return base64ClientData; + } + + /// <summary> + /// Tries to encrypt the specified data using the stored public key and store the encrypted data into + /// the output buffer. + /// </summary> + /// <param name="session"></param> + /// <param name="data">Data to encrypt</param> + /// <param name="outputBuffer">The buffer to store encrypted data in</param> + /// <returns> + /// The number of encrypted bytes written to the output buffer, + /// or false (0) if the operation failed, or if no credential is + /// stored. + /// </returns> + /// <exception cref="CryptographicException"></exception> + public static ERRNO TryEncryptClientData(this in SessionInfo session, ReadOnlySpan<byte> data, in Span<byte> outputBuffer) + { + if (!session.IsSet) + { + return false; + } + //try to get the public key from the client + string base64PubKey = session.GetBrowserPubKey(); + return TryEncryptClientData(base64PubKey, data, in outputBuffer); + } + /// <summary> + /// Tries to encrypt the specified data using the specified public key + /// </summary> + /// <param name="base64PubKey">A base64 encoded public key used to encrypt client data</param> + /// <param name="data">Data to encrypt</param> + /// <param name="outputBuffer">The buffer to store encrypted data in</param> + /// <returns> + /// The number of encrypted bytes written to the output buffer, + /// or false (0) if the operation failed, or if no credential is + /// specified. + /// </returns> + /// <exception cref="CryptographicException"></exception> + public static ERRNO TryEncryptClientData(ReadOnlySpan<char> base64PubKey, ReadOnlySpan<byte> data, in Span<byte> outputBuffer) + { + if (base64PubKey.IsEmpty) + { + return false; + } + //Alloc a buffer for decoding the public key + using UnsafeMemoryHandle<byte> pubKeyBuffer = Memory.UnsafeAlloc<byte>(PUBLIC_KEY_BUFFER_SIZE, true); + //Decode the public key + ERRNO pbkBytesWritten = VnEncoding.TryFromBase64Chars(base64PubKey, pubKeyBuffer); + //Try to encrypt the data + return pbkBytesWritten ? TryEncryptClientData(pubKeyBuffer.Span[..(int)pbkBytesWritten], data, in outputBuffer) : false; + } + /// <summary> + /// Tries to encrypt the specified data using the specified public key + /// </summary> + /// <param name="rawPubKey">The raw SKI public key</param> + /// <param name="data">Data to encrypt</param> + /// <param name="outputBuffer">The buffer to store encrypted data in</param> + /// <returns> + /// The number of encrypted bytes written to the output buffer, + /// or false (0) if the operation failed, or if no credential is + /// specified. + /// </returns> + /// <exception cref="CryptographicException"></exception> + public static ERRNO TryEncryptClientData(ReadOnlySpan<byte> rawPubKey, ReadOnlySpan<byte> data, in Span<byte> outputBuffer) + { + if (rawPubKey.IsEmpty) + { + return false; + } + //Setup new empty rsa + using RSA rsa = RSA.Create(); + //Import the public key + rsa.ImportSubjectPublicKeyInfo(rawPubKey, out _); + //Encrypt data with OaepSha256 as configured in the browser + return rsa.TryEncrypt(data, outputBuffer, ClientEncryptonPadding, out int bytesWritten) ? bytesWritten : false; + } + + /// <summary> + /// Stores the clients public key specified during login + /// </summary> + /// <param name="session"></param> + /// <param name="base64PubKey"></param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void SetBrowserPubKey(in SessionInfo session, string base64PubKey) => session[CLIENT_PUB_KEY_ENTRY] = base64PubKey; + + /// <summary> + /// Gets the clients stored public key that was specified during login + /// </summary> + /// <returns>The base64 encoded public key string specified at login</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetBrowserPubKey(this in SessionInfo session) => session[CLIENT_PUB_KEY_ENTRY]; + + /// <summary> + /// Stores the login key as a cookie in the current session as long as the session exists + /// </summary>/ + /// <param name="ev">The event to log-in</param> + /// <param name="localAccount">Does the session belong to a local user account</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void SetLogin(this HttpEntity ev, bool? localAccount = null) + { + //Make sure the session is loaded + if (!ev.Session.IsSet) + { + return; + } + string loginString = RandomHash.GetRandomBase64(LOGIN_COOKIE_SIZE); + //Set login cookie and session login hash + ev.Server.SetCookie(LOGIN_COOKIE_NAME, loginString, "", "/", LoginCookieLifespan, CookieSameSite.SameSite, true, true); + ev.Session.LoginHash = loginString; + //If not set get from session storage + localAccount ??= ev.Session.HasLocalAccount(); + //Set the client identifier cookie to a value indicating a local account + ev.Server.SetCookie(LOGIN_COOKIE_IDENTIFIER, localAccount.Value ? "1" : "2", "", "/", LoginCookieLifespan, CookieSameSite.SameSite, false, true); + } + + /// <summary> + /// Invalidates the login status of the current connection and session (if session is loaded) + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void InvalidateLogin(this HttpEntity ev) + { + //Expire the login cookie + ev.Server.ExpireCookie(LOGIN_COOKIE_NAME, sameSite: CookieSameSite.SameSite, secure: true); + //Expire the identifier cookie + ev.Server.ExpireCookie(LOGIN_COOKIE_IDENTIFIER, sameSite: CookieSameSite.SameSite, secure: true); + if (ev.Session.IsSet) + { + //Invalidate the session + ev.Session.Invalidate(); + } + } + + /// <summary> + /// Determines if the current session login cookie matches the value stored in the current session (if the session is loaded) + /// </summary> + /// <returns>True if the session is active, the cookie was properly received, and the cookie value matches the session. False otherwise</returns> + public static bool LoginCookieMatches(this HttpEntity ev) + { + //Sessions must be loaded + if (!ev.Session.IsSet) + { + return false; + } + //Try to get the login string from the request cookies + if (!ev.Server.RequestCookies.TryGetNonEmptyValue(LOGIN_COOKIE_NAME, out string? liCookie)) + { + return false; + } + /* + * Alloc buffer to do conversion and zero initial contents incase the + * payload size has been changed. + * + * Since the cookie size and the local copy should be the same size + * and equal to the LOGIN_COOKIE_SIZE constant, the buffer size should + * be 2 * LOGIN_COOKIE_SIZE, and it can be split in half and shared + * for both conversions + */ + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(2 * LOGIN_COOKIE_SIZE, true); + //Slice up buffers + Span<byte> cookieBuffer = buffer.Span[..LOGIN_COOKIE_SIZE]; + Span<byte> sessionBuffer = buffer.Span.Slice(LOGIN_COOKIE_SIZE, LOGIN_COOKIE_SIZE); + //Convert cookie and session hash value + if (Convert.TryFromBase64String(liCookie, cookieBuffer, out _) + && Convert.TryFromBase64String(ev.Session.LoginHash, sessionBuffer, out _)) + { + //Do a fixed time equal (probably overkill, but should not matter too much) + if(CryptographicOperations.FixedTimeEquals(cookieBuffer, sessionBuffer)) + { + //If the user is "logged in" and the request is using the POST method, then we can update the cookie + if(ev.Server.Method == HttpMethod.POST && ev.Session.Created.Add(RegenIdPeriod) < DateTimeOffset.UtcNow) + { + //Regen login token + ev.SetLogin(); + ev.Session.RegenID(); + } + + return true; + } + } + return false; + } + + /// <summary> + /// Determines if the client's login cookies need to be updated + /// to reflect its state with the current session's state + /// for the client + /// </summary> + /// <param name="ev"></param> + public static void ReconcileCookies(this HttpEntity ev) + { + //Only handle cookies if session is loaded and is a web based session + if (!ev.Session.IsSet || ev.Session.SessionType != SessionType.Web) + { + return; + } + if (ev.Session.IsNew) + { + //If either login cookies are set on a new session, clear them + if (ev.Server.RequestCookies.ContainsKey(LOGIN_COOKIE_NAME) || ev.Server.RequestCookies.ContainsKey(LOGIN_COOKIE_IDENTIFIER)) + { + //Expire the login cookie + ev.Server.ExpireCookie(LOGIN_COOKIE_NAME, sameSite:CookieSameSite.SameSite, secure:true); + //Expire the identifier cookie + ev.Server.ExpireCookie(LOGIN_COOKIE_IDENTIFIER, sameSite: CookieSameSite.SameSite, secure: true); + } + } + //If the session is not supposed to be logged in, clear the login cookies if they were set + else if (string.IsNullOrEmpty(ev.Session.LoginHash)) + { + //If one of either cookie is not set + if (ev.Server.RequestCookies.ContainsKey(LOGIN_COOKIE_NAME)) + { + //Expire the login cookie + ev.Server.ExpireCookie(LOGIN_COOKIE_NAME, sameSite: CookieSameSite.SameSite, secure: true); + } + if (ev.Server.RequestCookies.ContainsKey(LOGIN_COOKIE_IDENTIFIER)) + { + //Expire the identifier cookie + ev.Server.ExpireCookie(LOGIN_COOKIE_IDENTIFIER, sameSite: CookieSameSite.SameSite, secure: true); + } + } + } + + + /// <summary> + /// Stores the browser's id during a login process + /// </summary> + /// <param name="session"></param> + /// <param name="browserId">Browser id value to store</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void SetBrowserID(in SessionInfo session, string browserId) => session[BROWSER_ID_ENTRY] = browserId; + + /// <summary> + /// Gets the current browser's id if it was specified during login process + /// </summary> + /// <returns>The browser's id if set, <see cref="string.Empty"/> otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetBrowserID(this in SessionInfo session) => session[BROWSER_ID_ENTRY]; + + /// <summary> + /// Specifies that the current session belongs to a local user-account + /// </summary> + /// <param name="session"></param> + /// <param name="value">True for a local account, false otherwise</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void HasLocalAccount(this in SessionInfo session, bool value) => session[LOCAL_ACCOUNT_ENTRY] = value ? "1" : null; + /// <summary> + /// Gets a value indicating if the session belongs to a local user account + /// </summary> + /// <param name="session"></param> + /// <returns>True if the current user's account is a local account</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HasLocalAccount(this in SessionInfo session) => int.TryParse(session[LOCAL_ACCOUNT_ENTRY], out int value) && value > 0; + + #endregion + + #region Client Challenge + + /* + * Generates a secret that is used to compute the unique hmac digest of the + * current user's password. The digest is stored in the current session + * and used to compare future requests that require password re-authentication. + * The client will compute the digest of the user's password and send the digest + * instead of the user's password + */ + + /// <summary> + /// Generates a new password challenge for the current session and specified password + /// </summary> + /// <param name="session"></param> + /// <param name="password">The user's password to compute the hash of</param> + /// <returns>The raw derrivation key to send to the client</returns> + public static byte[] GenPasswordChallenge(this in SessionInfo session, PrivateString password) + { + ReadOnlySpan<char> rawPass = password; + //Calculate the password buffer size required + int passByteCount = Encoding.UTF8.GetByteCount(rawPass); + //Allocate the buffer + using UnsafeMemoryHandle<byte> bufferHandle = Memory.UnsafeAlloc<byte>(passByteCount + 64, true); + //Slice buffers + Span<byte> utf8PassBytes = bufferHandle.Span[..passByteCount]; + Span<byte> hashBuffer = bufferHandle.Span[passByteCount..]; + //Encode the password into the buffer + _ = Encoding.UTF8.GetBytes(rawPass, utf8PassBytes); + try + { + //Get random secret buffer + byte[] secretKey = RandomHash.GetRandomBytes(SESSION_CHALLENGE_SIZE); + //Compute the digest + int count = HMACSHA512.HashData(secretKey, utf8PassBytes, hashBuffer); + //Store the user's password digest + session[CHALLENGE_HMAC_ENTRY] = VnEncoding.ToBase32String(hashBuffer[..count], false); + return secretKey; + } + finally + { + //Wipe buffer + RandomHash.GetRandomBytes(utf8PassBytes); + } + } + /// <summary> + /// Verifies the stored unique digest of the user's password against + /// the client derrived password + /// </summary> + /// <param name="session"></param> + /// <param name="base64PasswordDigest">The base64 client derrived digest of the user's password to verify</param> + /// <returns>True if formatting was correct and the derrived passwords match, false otherwise</returns> + /// <exception cref="FormatException"></exception> + public static bool VerifyChallenge(this in SessionInfo session, ReadOnlySpan<char> base64PasswordDigest) + { + string base32Digest = session[CHALLENGE_HMAC_ENTRY]; + if (string.IsNullOrWhiteSpace(base32Digest)) + { + return false; + } + int bufSize = base32Digest.Length + base64PasswordDigest.Length; + //Alloc buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(bufSize); + //Split buffers + Span<byte> localBuf = buffer.Span[..base32Digest.Length]; + Span<byte> passBuf = buffer.Span[base32Digest.Length..]; + //Recover the stored base32 digest + ERRNO count = VnEncoding.TryFromBase32Chars(base32Digest, localBuf); + if (!count) + { + return false; + } + //Recover base64 bytes + if(!Convert.TryFromBase64Chars(base64PasswordDigest, passBuf, out int passBytesWritten)) + { + return false; + } + //Trim buffers + localBuf = localBuf[..(int)count]; + passBuf = passBuf[..passBytesWritten]; + //Compare and return + return CryptographicOperations.FixedTimeEquals(passBuf, localBuf); + } + + #endregion + + #region Privilage Extensions + /// <summary> + /// Compares the users privilage level against the specified level + /// </summary> + /// <param name="session"></param> + /// <param name="level">64bit privilage level to compare</param> + /// <returns>true if the current user has at least the specified level or higher</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HasLevel(this in SessionInfo session, byte level) => (session.Privilages & LEVEL_MSK) >= (((ulong)level << LEVEL_MSK_OFFSET) & LEVEL_MSK); + /// <summary> + /// Determines if the group ID of the current user matches the specified group + /// </summary> + /// <param name="session"></param> + /// <param name="groupId">Group ID to compare</param> + /// <returns>true if the user belongs to the group, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HasGroup(this in SessionInfo session, ushort groupId) => (session.Privilages & GROUP_MSK) == (((ulong)groupId << GROUP_MSK_OFFSET) & GROUP_MSK); + /// <summary> + /// Determines if the current user has an equivalent option code + /// </summary> + /// <param name="session"></param> + /// <param name="option">Option code check</param> + /// <returns>true if the user options field equals the option</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HasOption(this in SessionInfo session, byte option) => (session.Privilages & OPTIONS_MSK) == (((ulong)option << OPTIONS_MSK_OFFSET) & OPTIONS_MSK); + + /// <summary> + /// Returns the status of the user's privlage read bit + /// </summary> + /// <returns>true if the current user has the read permission, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool CanRead(this in SessionInfo session) => (session.Privilages & READ_MSK) == READ_MSK; + /// <summary> + /// Returns the status of the user's privlage write bit + /// </summary> + /// <returns>true if the current user has the write permission, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool CanWrite(this in SessionInfo session) => (session.Privilages & WRITE_MSK) == WRITE_MSK; + /// <summary> + /// Returns the status of the user's privlage delete bit + /// </summary> + /// <returns>true if the current user has the delete permission, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool CanDelete(this in SessionInfo session) => (session.Privilages & DELETE_MSK) == DELETE_MSK; + #endregion + + #region flc + + /// <summary> + /// Gets the current number of failed login attempts + /// </summary> + /// <param name="user"></param> + /// <returns>The current number of failed login attempts</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static TimestampedCounter FailedLoginCount(this IUser user) + { + ulong value = user.GetValueType<string, ulong>(FAILED_LOGIN_ENTRY); + return (TimestampedCounter)value; + } + /// <summary> + /// Sets the number of failed login attempts for the current session + /// </summary> + /// <param name="user"></param> + /// <param name="value">The value to set the failed login attempt count</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void FailedLoginCount(this IUser user, uint value) + { + TimestampedCounter counter = new(value); + //Cast the counter to a ulong and store as a ulong + user.SetValueType(FAILED_LOGIN_ENTRY, (ulong)counter); + } + /// <summary> + /// Sets the number of failed login attempts for the current session + /// </summary> + /// <param name="user"></param> + /// <param name="value">The value to set the failed login attempt count</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void FailedLoginCount(this IUser user, TimestampedCounter value) + { + //Cast the counter to a ulong and store as a ulong + user.SetValueType(FAILED_LOGIN_ENTRY, (ulong)value); + } + /// <summary> + /// Increments the failed login attempt count + /// </summary> + /// <param name="user"></param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void FailedLoginIncrement(this IUser user) + { + TimestampedCounter current = user.FailedLoginCount(); + user.FailedLoginCount(current.Count + 1); + } + + #endregion + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Accounts/INonce.cs b/Plugins.Essentials/src/Accounts/INonce.cs new file mode 100644 index 0000000..7d53183 --- /dev/null +++ b/Plugins.Essentials/src/Accounts/INonce.cs @@ -0,0 +1,90 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: INonce.cs +* +* INonce.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils; +using VNLib.Utils.Memory; + +namespace VNLib.Plugins.Essentials.Accounts +{ + /// <summary> + /// Represents a object that performs storage and computation of nonce values + /// </summary> + public interface INonce + { + /// <summary> + /// Generates a random nonce for the current instance and + /// returns a base32 encoded string. + /// </summary> + /// <param name="buffer">The buffer to write a copy of the nonce value to</param> + void ComputeNonce(Span<byte> buffer); + /// <summary> + /// Compares the raw nonce bytes to the current nonce to determine + /// if the supplied nonce value is valid + /// </summary> + /// <param name="nonceBytes">The binary value of the nonce</param> + /// <returns>True if the nonce values are equal, flase otherwise</returns> + bool VerifyNonce(ReadOnlySpan<byte> nonceBytes); + } + + /// <summary> + /// Provides INonce extensions for computing/verifying nonce values + /// </summary> + public static class NonceExtensions + { + /// <summary> + /// Computes a base32 nonce of the specified size and returns a string + /// representation + /// </summary> + /// <param name="nonce"></param> + /// <param name="size">The size (in bytes) of the nonce</param> + /// <returns>The base32 string of the computed nonce</returns> + public static string ComputeNonce<T>(this T nonce, int size) where T: INonce + { + //Alloc bin buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(size); + //Compute nonce + nonce.ComputeNonce(buffer.Span); + //Return base32 string + return VnEncoding.ToBase32String(buffer.Span, false); + } + /// <summary> + /// Compares the base32 encoded nonce value against the previously + /// generated nonce + /// </summary> + /// <param name="nonce"></param> + /// <param name="base32Nonce">The base32 encoded nonce string</param> + /// <returns>True if the nonce values are equal, flase otherwise</returns> + public static bool VerifyNonce<T>(this T nonce, ReadOnlySpan<char> base32Nonce) where T : INonce + { + //Alloc bin buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(base32Nonce.Length); + //Decode base32 nonce + ERRNO count = VnEncoding.TryFromBase32Chars(base32Nonce, buffer.Span); + //Verify nonce + return nonce.VerifyNonce(buffer.Span[..(int)count]); + } + } +} diff --git a/Plugins.Essentials/src/Accounts/LoginMessage.cs b/Plugins.Essentials/src/Accounts/LoginMessage.cs new file mode 100644 index 0000000..ebc616e --- /dev/null +++ b/Plugins.Essentials/src/Accounts/LoginMessage.cs @@ -0,0 +1,102 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: LoginMessage.cs +* +* LoginMessage.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json.Serialization; +using VNLib.Utils.Memory; + +namespace VNLib.Plugins.Essentials.Accounts +{ + /// <summary> + /// A uniform JSON login message for the + /// accounts provider to use + /// </summary> + /// <remarks> + /// NOTE: This class derrives from <see cref="PrivateStringManager"/> + /// and should be disposed properly + /// </remarks> + public class LoginMessage : PrivateStringManager + { + /// <summary> + /// A property + /// </summary> + [JsonPropertyName("username")] + public string UserName { get; set; } + /// <summary> + /// A protected string property that + /// may represent a user's password + /// </summary> + [JsonPropertyName("password")] + public string Password + { + get => base[0]; + set => base[0] = value; + } + [JsonPropertyName("localtime")] + public string Lt + { + get => LocalTime.ToString("O"); + //Try to parse the supplied time string, and use the datetime.min if the time string is invalid + set => LocalTime = DateTimeOffset.TryParse(value, out DateTimeOffset local) ? local : DateTimeOffset.MinValue; + } + + /// <summary> + /// Represents the clients local time in a <see cref="DateTime"/> struct + /// </summary> + [JsonIgnore] + public DateTimeOffset LocalTime { get; set; } + /// <summary> + /// The clients specified local-language + /// </summary> + [JsonPropertyName("locallanguage")] + public string LocalLanguage { get; set; } + /// <summary> + /// The clients shared public key used for encryption, this property is not protected + /// </summary> + [JsonPropertyName("pubkey")] + public string ClientPublicKey { get; set; } + /// <summary> + /// The clients browser id if shared + /// </summary> + [JsonPropertyName("clientid")] + public string ClientID { get; set; } + /// <summary> + /// Initailzies a new <see cref="LoginMessage"/> and its parent <see cref="PrivateStringManager"/> + /// base + /// </summary> + public LoginMessage() : this(1) { } + /// <summary> + /// Allows for derrives classes to have multple protected + /// string elements + /// </summary> + /// <param name="protectedElementSize"> + /// The number of procted string elements required + /// </param> + /// <remarks> + /// NOTE: <paramref name="protectedElementSize"/> must be at-least 1 + /// or access to <see cref="Password"/> will throw + /// </remarks> + protected LoginMessage(int protectedElementSize = 1) : base(protectedElementSize) { } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Accounts/PasswordHashing.cs b/Plugins.Essentials/src/Accounts/PasswordHashing.cs new file mode 100644 index 0000000..1c3770b --- /dev/null +++ b/Plugins.Essentials/src/Accounts/PasswordHashing.cs @@ -0,0 +1,244 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: PasswordHashing.cs +* +* PasswordHashing.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Security.Cryptography; + +using VNLib.Hashing; +using VNLib.Utils; +using VNLib.Utils.Memory; + +namespace VNLib.Plugins.Essentials.Accounts +{ + /// <summary> + /// A delegate method to recover a temporary copy of the secret/pepper + /// for a request + /// </summary> + /// <param name="buffer">The buffer to write the pepper to</param> + /// <returns>The number of bytes written to the buffer</returns> + public delegate ERRNO SecretAction(Span<byte> buffer); + + /// <summary> + /// Provides a structrued password hashing system implementing the <seealso cref="VnArgon2"/> library + /// with fixed time comparison + /// </summary> + public sealed class PasswordHashing + { + private readonly SecretAction _getter; + private readonly int _secretSize; + + private readonly uint TimeCost; + private readonly uint MemoryCost; + private readonly uint HashLen; + private readonly int SaltLen; + private readonly uint Parallelism; + + /// <summary> + /// Initalizes the <see cref="PasswordHashing"/> class + /// </summary> + /// <param name="getter"></param> + /// <param name="secreteSize">The expected size of the secret (the size of the buffer to alloc for a copy)</param> + /// <param name="saltLen">A positive integer for the size of the random salt used during the hashing proccess</param> + /// <param name="timeCost">The Argon2 time cost parameter</param> + /// <param name="memoryCost">The Argon2 memory cost parameter</param> + /// <param name="hashLen">The size of the hash to produce during hashing operations</param> + /// <param name="parallism"> + /// The Argon2 parallelism parameter (the number of threads to use for hasing) + /// (default = 0 - the number of processors) + /// </param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public PasswordHashing(SecretAction getter, int secreteSize, int saltLen = 32, uint timeCost = 4, uint memoryCost = UInt16.MaxValue, uint parallism = 0, uint hashLen = 128) + { + //Store getter + _getter = getter ?? throw new ArgumentNullException(nameof(getter)); + _secretSize = secreteSize; + + //Store parameters + HashLen = hashLen; + //Store maginitude as a unit + MemoryCost = memoryCost; + TimeCost = timeCost; + SaltLen = saltLen; + Parallelism = parallism < 1 ? (uint)Environment.ProcessorCount : parallism; + } + + /// <summary> + /// Verifies a password against its previously encoded hash. + /// </summary> + /// <param name="passHash">Previously hashed password</param> + /// <param name="password">Raw password to compare against</param> + /// <returns>true if bytes derrived from password match the hash, false otherwise</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="VnArgon2PasswordFormatException"></exception> + public bool Verify(PrivateString passHash, PrivateString password) + { + //Casting PrivateStrings to spans will reference the base string directly + return Verify((ReadOnlySpan<char>)passHash, (ReadOnlySpan<char>)password); + } + /// <summary> + /// Verifies a password against its previously encoded hash. + /// </summary> + /// <param name="passHash">Previously hashed password</param> + /// <param name="password">Raw password to compare against</param> + /// <returns>true if bytes derrived from password match the hash, false otherwise</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="VnArgon2Exception"></exception> + /// <exception cref="VnArgon2PasswordFormatException"></exception> + public bool Verify(ReadOnlySpan<char> passHash, ReadOnlySpan<char> password) + { + if(passHash.IsEmpty || password.IsEmpty) + { + return false; + } + //alloc secret buffer + using UnsafeMemoryHandle<byte> secretBuffer = Memory.UnsafeAlloc<byte>(_secretSize, true); + try + { + //Get the secret from the callback + ERRNO count = _getter(secretBuffer.Span); + //Verify + return VnArgon2.Verify2id(password, passHash, secretBuffer.Span[..(int)count]); + } + finally + { + //Erase secret buffer + Memory.InitializeBlock(secretBuffer.Span); + } + } + /// <summary> + /// Verifies a password against its hash. Partially exposes the Argon2 api. + /// </summary> + /// <param name="hash">Previously hashed password</param> + /// <param name="salt">The salt used to hash the original password</param> + /// <param name="password">The password to hash and compare against </param> + /// <returns>true if bytes derrived from password match the hash, false otherwise</returns> + /// <exception cref="VnArgon2Exception"></exception> + /// <remarks>Uses fixed time comparison from <see cref="CryptographicOperations"/> class</remarks> + public bool Verify(ReadOnlySpan<byte> hash, ReadOnlySpan<byte> salt, ReadOnlySpan<byte> password) + { + //Alloc a buffer with the same size as the hash + using UnsafeMemoryHandle<byte> hashBuf = Memory.UnsafeAlloc<byte>(hash.Length, true); + //Hash the password with the current config + Hash(password, salt, hashBuf.Span); + //Compare the hashed password to the specified hash and return results + return CryptographicOperations.FixedTimeEquals(hash, hashBuf.Span); + } + + /// <summary> + /// Hashes a specified password, with the initialized pepper, and salted with CNG random bytes. + /// </summary> + /// <param name="password">Password to be hashed</param> + /// <exception cref="VnArgon2Exception"></exception> + /// <returns>A <see cref="PrivateString"/> of the hashed and encoded password</returns> + public PrivateString Hash(PrivateString password) => Hash((ReadOnlySpan<char>)password); + + /// <summary> + /// Hashes a specified password, with the initialized pepper, and salted with CNG random bytes. + /// </summary> + /// <param name="password">Password to be hashed</param> + /// <exception cref="VnArgon2Exception"></exception> + /// <returns>A <see cref="PrivateString"/> of the hashed and encoded password</returns> + public PrivateString Hash(ReadOnlySpan<char> password) + { + //Alloc shared buffer for the salt and secret buffer + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(SaltLen + _secretSize, true); + try + { + //Split buffers + Span<byte> saltBuf = buffer.Span[..SaltLen]; + Span<byte> secretBuf = buffer.Span[SaltLen..]; + + //Fill the buffer with random bytes + RandomHash.GetRandomBytes(saltBuf); + + //recover the secret + ERRNO count = _getter(secretBuf); + + //Hashes a password, with the current parameters + return (PrivateString)VnArgon2.Hash2id(password, saltBuf, secretBuf[..(int)count], TimeCost, MemoryCost, Parallelism, HashLen); + } + finally + { + Memory.InitializeBlock(buffer.Span); + } + } + + /// <summary> + /// Hashes a specified password, with the initialized pepper, and salted with a CNG random bytes. + /// </summary> + /// <param name="password">Password to be hashed</param> + /// <exception cref="VnArgon2Exception"></exception> + /// <returns>A <see cref="PrivateString"/> of the hashed and encoded password</returns> + public PrivateString Hash(ReadOnlySpan<byte> password) + { + using UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(SaltLen + _secretSize, true); + try + { + //Split buffers + Span<byte> saltBuf = buffer.Span[..SaltLen]; + Span<byte> secretBuf = buffer.Span[SaltLen..]; + + //Fill the buffer with random bytes + RandomHash.GetRandomBytes(saltBuf); + + //recover the secret + ERRNO count = _getter(secretBuf); + + //Hashes a password, with the current parameters + return (PrivateString)VnArgon2.Hash2id(password, saltBuf, secretBuf[..(int)count], TimeCost, MemoryCost, Parallelism, HashLen); + } + finally + { + Memory.InitializeBlock(buffer.Span); + } + } + /// <summary> + /// Partially exposes the Argon2 api. Hashes the specified password, with the initialized pepper. + /// Writes the raw hash output to the specified buffer + /// </summary> + /// <param name="password">Password to be hashed</param> + /// <param name="salt">Salt to hash the password with</param> + /// <param name="hashOutput">The output buffer to store the hashed password to. The exact length of this buffer is the hash size</param> + /// <exception cref="VnArgon2Exception"></exception> + public void Hash(ReadOnlySpan<byte> password, ReadOnlySpan<byte> salt, Span<byte> hashOutput) + { + //alloc secret buffer + using UnsafeMemoryHandle<byte> secretBuffer = Memory.UnsafeAlloc<byte>(_secretSize, true); + try + { + //Get the secret from the callback + ERRNO count = _getter(secretBuffer.Span); + //Hashes a password, with the current parameters + VnArgon2.Hash2id(password, salt, secretBuffer.Span[..(int)count], hashOutput, TimeCost, MemoryCost, Parallelism); + } + finally + { + //Erase secret buffer + Memory.InitializeBlock(secretBuffer.Span); + } + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Content/IPageRouter.cs b/Plugins.Essentials/src/Content/IPageRouter.cs new file mode 100644 index 0000000..e6952f4 --- /dev/null +++ b/Plugins.Essentials/src/Content/IPageRouter.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: IPageRouter.cs +* +* IPageRouter.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Threading.Tasks; + +//Import account system for privilage masks + +namespace VNLib.Plugins.Essentials.Content +{ + /// <summary> + /// Determines file routines (routing) for incomming connections + /// </summary> + public interface IPageRouter + { + /// <summary> + /// Determines what file path to return to a user for the given incoming connection + /// </summary> + /// <param name="entity">The connection to proccess</param> + /// <returns>A <see cref="ValueTask"/> that returns the <see cref="FileProcessArgs"/> to pass to the file processor</returns> + ValueTask<FileProcessArgs> RouteAsync(HttpEntity entity); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Endpoints/ProtectedWebEndpoint.cs b/Plugins.Essentials/src/Endpoints/ProtectedWebEndpoint.cs new file mode 100644 index 0000000..bced960 --- /dev/null +++ b/Plugins.Essentials/src/Endpoints/ProtectedWebEndpoint.cs @@ -0,0 +1,58 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ProtectedWebEndpoint.cs +* +* ProtectedWebEndpoint.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; + +using VNLib.Utils; +using VNLib.Plugins.Essentials.Accounts; + +namespace VNLib.Plugins.Essentials.Endpoints +{ + /// <summary> + /// Implements <see cref="UnprotectedWebEndpoint"/> to provide + /// authoriation checks before processing + /// </summary> + public abstract class ProtectedWebEndpoint : UnprotectedWebEndpoint + { + ///<inheritdoc/> + protected override ERRNO PreProccess(HttpEntity entity) + { + if (!base.PreProccess(entity)) + { + return false; + } + //The loggged in flag must be set, and the token must also match + if (!entity.LoginCookieMatches() || !entity.TokenMatches()) + { + //Return unauthorized status + entity.CloseResponse(HttpStatusCode.Unauthorized); + //A return value less than 0 signals a virtual skip event + return -1; + } + //Continue + return true; + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Endpoints/ProtectionSettings.cs b/Plugins.Essentials/src/Endpoints/ProtectionSettings.cs new file mode 100644 index 0000000..77620ac --- /dev/null +++ b/Plugins.Essentials/src/Endpoints/ProtectionSettings.cs @@ -0,0 +1,103 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ProtectionSettings.cs +* +* ProtectionSettings.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Plugins.Essentials.Endpoints +{ + /// <summary> + /// A data structure containing a basic security protocol + /// for connection pre-checks. Settings are the most + /// strict by default + /// </summary> + public readonly struct ProtectionSettings : IEquatable<ProtectionSettings> + { + /// <summary> + /// Requires TLS be enabled for all incomming requets (or loopback adapter) + /// </summary> + public readonly bool DisabledTlsRequired { get; init; } + + /// <summary> + /// Checks that sessions are enabled for incomming requests + /// and that they are not new sessions. + /// </summary> + public readonly bool DisableSessionsRequired { get; init; } + + /// <summary> + /// Allows connections that define cross-site sec headers + /// to be processed or denied (denied by default) + /// </summary> + public readonly bool DisableCrossSiteDenied { get; init; } + + /// <summary> + /// Enables referr match protection. Requires that if a referer header is + /// set that it matches the current origin + /// </summary> + public readonly bool DisableRefererMatch { get; init; } + + /// <summary> + /// Requires all connections to have pass an IsBrowser() check + /// (requires a valid user-agent header that contains Mozilla in + /// the string) + /// </summary> + public readonly bool DisableBrowsersOnly { get; init; } + + /// <summary> + /// If the connection has a valid session, verifies that the + /// stored session origin matches the client's origin header. + /// (confirms the session is coming from the same origin it + /// was created on) + /// </summary> + public readonly bool DisableVerifySessionCors { get; init; } + + /// <summary> + /// Disables response caching, by setting the cache control headers appropriatly. + /// Default is disabled + /// </summary> + public readonly bool EnableCaching { get; init; } + + + ///<inheritdoc/> + public override bool Equals(object obj) => obj is ProtectionSettings settings && Equals(settings); + ///<inheritdoc/> + public override int GetHashCode() => base.GetHashCode(); + + ///<inheritdoc/> + public static bool operator ==(ProtectionSettings left, ProtectionSettings right) => left.Equals(right); + ///<inheritdoc/> + public static bool operator !=(ProtectionSettings left, ProtectionSettings right) => !(left == right); + + ///<inheritdoc/> + public bool Equals(ProtectionSettings other) + { + return DisabledTlsRequired == other.DisabledTlsRequired && + DisableSessionsRequired == other.DisableSessionsRequired && + DisableCrossSiteDenied == other.DisableCrossSiteDenied && + DisableRefererMatch == other.DisableRefererMatch && + DisableBrowsersOnly == other.DisableBrowsersOnly && + DisableVerifySessionCors == other.DisableVerifySessionCors && + EnableCaching == other.EnableCaching; + } + } +} diff --git a/Plugins.Essentials/src/Endpoints/ResourceEndpointBase.cs b/Plugins.Essentials/src/Endpoints/ResourceEndpointBase.cs new file mode 100644 index 0000000..4af3c30 --- /dev/null +++ b/Plugins.Essentials/src/Endpoints/ResourceEndpointBase.cs @@ -0,0 +1,346 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ResourceEndpointBase.cs +* +* ResourceEndpointBase.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Net.Sockets; +using System.Threading.Tasks; + +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.Logging; +using VNLib.Plugins.Essentials.Extensions; + +namespace VNLib.Plugins.Essentials.Endpoints +{ + + /// <summary> + /// Provides a base class for implementing un-authenticated resource endpoints + /// with basic (configurable) security checks + /// </summary> + public abstract class ResourceEndpointBase : VirtualEndpoint<HttpEntity> + { + /// <summary> + /// Default protection settings. Protection settings are the most + /// secure by default, should be loosened an necessary + /// </summary> + protected virtual ProtectionSettings EndpointProtectionSettings { get; } + + ///<inheritdoc/> + public override async ValueTask<VfReturnType> Process(HttpEntity entity) + { + try + { + ERRNO preProc = PreProccess(entity); + if (preProc == ERRNO.E_FAIL) + { + return VfReturnType.Forbidden; + } + //Entity was responded to by the pre-processor + if (preProc < 0) + { + return VfReturnType.VirtualSkip; + } + //If websockets are quested allow them to be processed in a logged-in/secure context + if (entity.Server.IsWebSocketRequest) + { + return await WebsocketRequestedAsync(entity); + } + ValueTask<VfReturnType> op = entity.Server.Method switch + { + //Get request to get account + HttpMethod.GET => GetAsync(entity), + HttpMethod.POST => PostAsync(entity), + HttpMethod.DELETE => DeleteAsync(entity), + HttpMethod.PUT => PutAsync(entity), + HttpMethod.PATCH => PatchAsync(entity), + HttpMethod.OPTIONS => OptionsAsync(entity), + _ => AlternateMethodAsync(entity, entity.Server.Method), + }; + return await op; + } + catch (InvalidJsonRequestException ije) + { + //Write the je to debug log + Log.Debug(ije, "Failed to de-serialize a request entity body"); + //If the method is not POST/PUT/PATCH return a json message + if ((entity.Server.Method & (HttpMethod.HEAD | HttpMethod.OPTIONS | HttpMethod.TRACE | HttpMethod.DELETE)) > 0) + { + return VfReturnType.BadRequest; + } + //Only allow if json is an accepted response type + if (!entity.Server.Accepts(ContentType.Json)) + { + return VfReturnType.BadRequest; + } + //Build web-message + WebMessage webm = new() + { + Result = "Request body is not valid json" + }; + //Set the response webm + entity.CloseResponseJson(HttpStatusCode.BadRequest, webm); + //return virtual + return VfReturnType.VirtualSkip; + } + catch (TerminateConnectionException) + { + //A TC exception is intentional and should always propagate to the runtime + throw; + } + catch (ContentTypeUnacceptableException) + { + /* + * The runtime will handle a 406 unaccetptable response + * and invoke the proper error app handler + */ + throw; + } + //Re-throw exceptions that are cause by reading the transport layer + catch (IOException ioe) when (ioe.InnerException is SocketException) + { + throw; + } + catch (Exception ex) + { + //Log an uncaught excetpion and return an error code (log may not be initialized) + Log?.Error(ex); + return VfReturnType.Error; + } + } + + /// <summary> + /// Allows for synchronous Pre-Processing of an entity. The result + /// will determine if the method processing methods will be invoked, or + /// a <see cref="VfReturnType.Forbidden"/> error code will be returned + /// </summary> + /// <param name="entity">The incomming request to process</param> + /// <returns> + /// True if processing should continue, false if the response should be + /// <see cref="VfReturnType.Forbidden"/>, less than 0 if entity was + /// responded to. + /// </returns> + protected virtual ERRNO PreProccess(HttpEntity entity) + { + //Disable cache if requested + if (!EndpointProtectionSettings.EnableCaching) + { + entity.Server.SetNoCache(); + } + + //Enforce TLS + if (!EndpointProtectionSettings.DisabledTlsRequired && !entity.IsSecure && !entity.IsLocalConnection) + { + return false; + } + + //Enforce browser check + if (!EndpointProtectionSettings.DisableBrowsersOnly && !entity.Server.IsBrowser()) + { + return false; + } + + //Enforce refer check + if (!EndpointProtectionSettings.DisableRefererMatch && entity.Server.Referer != null && !entity.Server.RefererMatch()) + { + return false; + } + + //enforce session basic + if (!EndpointProtectionSettings.DisableSessionsRequired && (!entity.Session.IsSet || entity.Session.IsNew)) + { + return false; + } + + /* + * If sessions are required, verify cors is set, and the client supplied an origin header, + * verify that it matches the origin that was specified during session initialization + */ + if ((!EndpointProtectionSettings.DisableSessionsRequired & !EndpointProtectionSettings.DisableVerifySessionCors) && entity.Server.Origin != null && !entity.Session.CrossOriginMatch) + { + return false; + } + + //Enforce cross-site + if (!EndpointProtectionSettings.DisableCrossSiteDenied && entity.Server.IsCrossSite()) + { + return false; + } + + return true; + } + + /// <summary> + /// This method gets invoked when an incoming POST request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual ValueTask<VfReturnType> PostAsync(HttpEntity entity) + { + return ValueTask.FromResult(Post(entity)); + } + /// <summary> + /// This method gets invoked when an incoming GET request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual ValueTask<VfReturnType> GetAsync(HttpEntity entity) + { + return ValueTask.FromResult(Get(entity)); + } + /// <summary> + /// This method gets invoked when an incoming DELETE request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual ValueTask<VfReturnType> DeleteAsync(HttpEntity entity) + { + return ValueTask.FromResult(Delete(entity)); + } + /// <summary> + /// This method gets invoked when an incoming PUT request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual ValueTask<VfReturnType> PutAsync(HttpEntity entity) + { + return ValueTask.FromResult(Put(entity)); + } + /// <summary> + /// This method gets invoked when an incoming PATCH request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual ValueTask<VfReturnType> PatchAsync(HttpEntity entity) + { + return ValueTask.FromResult(Patch(entity)); + } + + protected virtual ValueTask<VfReturnType> OptionsAsync(HttpEntity entity) + { + return ValueTask.FromResult(Options(entity)); + } + + /// <summary> + /// Invoked when a request is received for a method other than GET, POST, DELETE, or PUT; + /// </summary> + /// <param name="entity">The entity that </param> + /// <param name="method">The request method</param> + /// <returns>The results of the processing</returns> + protected virtual ValueTask<VfReturnType> AlternateMethodAsync(HttpEntity entity, HttpMethod method) + { + return ValueTask.FromResult(AlternateMethod(entity, method)); + } + + /// <summary> + /// Invoked when the current endpoint received a websocket request + /// </summary> + /// <param name="entity">The entity that requested the websocket</param> + /// <returns>The results of the operation</returns> + protected virtual ValueTask<VfReturnType> WebsocketRequestedAsync(HttpEntity entity) + { + return ValueTask.FromResult(WebsocketRequested(entity)); + } + + /// <summary> + /// This method gets invoked when an incoming POST request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual VfReturnType Post(HttpEntity entity) + { + //Return method not allowed + entity.CloseResponse(HttpStatusCode.MethodNotAllowed); + return VfReturnType.VirtualSkip; + } + /// <summary> + /// This method gets invoked when an incoming GET request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual VfReturnType Get(HttpEntity entity) + { + return VfReturnType.ProcessAsFile; + } + /// <summary> + /// This method gets invoked when an incoming DELETE request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual VfReturnType Delete(HttpEntity entity) + { + entity.CloseResponse(HttpStatusCode.MethodNotAllowed); + return VfReturnType.VirtualSkip; + } + /// <summary> + /// This method gets invoked when an incoming PUT request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual VfReturnType Put(HttpEntity entity) + { + entity.CloseResponse(HttpStatusCode.MethodNotAllowed); + return VfReturnType.VirtualSkip; + } + /// <summary> + /// This method gets invoked when an incoming PATCH request to the endpoint has been requested. + /// </summary> + /// <param name="entity">The entity to be processed</param> + /// <returns>The result of the operation to return to the file processor</returns> + protected virtual VfReturnType Patch(HttpEntity entity) + { + entity.CloseResponse(HttpStatusCode.MethodNotAllowed); + return VfReturnType.VirtualSkip; + } + /// <summary> + /// Invoked when a request is received for a method other than GET, POST, DELETE, or PUT; + /// </summary> + /// <param name="entity">The entity that </param> + /// <param name="method">The request method</param> + /// <returns>The results of the processing</returns> + protected virtual VfReturnType AlternateMethod(HttpEntity entity, HttpMethod method) + { + //Return method not allowed + entity.CloseResponse(HttpStatusCode.MethodNotAllowed); + return VfReturnType.VirtualSkip; + } + + protected virtual VfReturnType Options(HttpEntity entity) + { + return VfReturnType.Forbidden; + } + + /// <summary> + /// Invoked when the current endpoint received a websocket request + /// </summary> + /// <param name="entity">The entity that requested the websocket</param> + /// <returns>The results of the operation</returns> + protected virtual VfReturnType WebsocketRequested(HttpEntity entity) + { + entity.CloseResponse(HttpStatusCode.Forbidden); + return VfReturnType.VirtualSkip; + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Endpoints/UnprotectedWebEndpoint.cs b/Plugins.Essentials/src/Endpoints/UnprotectedWebEndpoint.cs new file mode 100644 index 0000000..cc923c7 --- /dev/null +++ b/Plugins.Essentials/src/Endpoints/UnprotectedWebEndpoint.cs @@ -0,0 +1,42 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UnprotectedWebEndpoint.cs +* +* UnprotectedWebEndpoint.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using VNLib.Utils; + +namespace VNLib.Plugins.Essentials.Endpoints +{ + /// <summary> + /// A base class for un-authenticated web (browser) based resource endpoints + /// to implement. Adds additional security checks + /// </summary> + public abstract class UnprotectedWebEndpoint : ResourceEndpointBase + { + ///<inheritdoc/> + protected override ERRNO PreProccess(HttpEntity entity) + { + return base.PreProccess(entity) + && (!entity.Session.IsSet || entity.Session.SessionType == Sessions.SessionType.Web); + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Endpoints/VirtualEndpoint.cs b/Plugins.Essentials/src/Endpoints/VirtualEndpoint.cs new file mode 100644 index 0000000..5beb4b9 --- /dev/null +++ b/Plugins.Essentials/src/Endpoints/VirtualEndpoint.cs @@ -0,0 +1,67 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: VirtualEndpoint.cs +* +* VirtualEndpoint.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading.Tasks; + +using VNLib.Utils.Logging; + +namespace VNLib.Plugins.Essentials.Endpoints +{ + /// <summary> + /// Provides a base class for <see cref="IVirtualEndpoint{T}"/> entity processors + /// with checks and a log provider + /// </summary> + /// <typeparam name="T">The entity type to process</typeparam> + public abstract class VirtualEndpoint<T> : MarshalByRefObject, IVirtualEndpoint<T> + { + ///<inheritdoc/> + public virtual string Path { get; protected set; } + + /// <summary> + /// An <see cref="ILogProvider"/> to write logs to + /// </summary> + protected ILogProvider Log { get; private set; } + + /// <summary> + /// Sets the log and path and checks the values + /// </summary> + /// <param name="Path">The path this instance represents</param> + /// <param name="log">The log provider that will be used</param> + /// <exception cref="ArgumentException"></exception> + protected void InitPathAndLog(string Path, ILogProvider log) + { + if (string.IsNullOrWhiteSpace(Path) || Path[0] != '/') + { + throw new ArgumentException("Path must begin with a '/' character", nameof(Path)); + } + //Store path + this.Path = Path; + //Store log + Log = log ?? throw new ArgumentNullException(nameof(log)); + } + ///<inheritdoc/> + public abstract ValueTask<VfReturnType> Process(T entity); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/EventProcessor.cs b/Plugins.Essentials/src/EventProcessor.cs new file mode 100644 index 0000000..826ed00 --- /dev/null +++ b/Plugins.Essentials/src/EventProcessor.cs @@ -0,0 +1,728 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: EventProcessor.cs +* +* EventProcessor.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Linq; +using System.Threading; +using System.Net.Sockets; +using System.Threading.Tasks; +using System.Collections.Generic; + +using VNLib.Net; +using VNLib.Net.Http; +using VNLib.Utils.IO; +using VNLib.Utils.Logging; +using VNLib.Utils.Resources; +using VNLib.Plugins.Essentials.Content; +using VNLib.Plugins.Essentials.Sessions; +using VNLib.Plugins.Essentials.Extensions; + +#nullable enable + +namespace VNLib.Plugins.Essentials +{ + + /// <summary> + /// Provides an abstract base implementation of <see cref="IWebRoot"/> + /// that breaks down simple processing procedures, routing, and session + /// loading. + /// </summary> + public abstract class EventProcessor : IWebRoot + { + private static readonly AsyncLocal<EventProcessor?> _currentProcessor = new(); + + /// <summary> + /// Gets the current (ambient) async local event processor + /// </summary> + public static EventProcessor? Current => _currentProcessor.Value; + + /// <summary> + /// The filesystem entrypoint path for the site + /// </summary> + public abstract string Directory { get; } + ///<inheritdoc/> + public abstract string Hostname { get; } + + /// <summary> + /// Gets the EP processing options + /// </summary> + public abstract IEpProcessingOptions Options { get; } + + /// <summary> + /// Event log provider + /// </summary> + protected abstract ILogProvider Log { get; } + + /// <summary> + /// <para> + /// Called when the server intends to process a file and requires translation from a + /// uri path to a usable filesystem path + /// </para> + /// <para> + /// NOTE: This function must be thread-safe! + /// </para> + /// </summary> + /// <param name="requestPath">The path requested by the request </param> + /// <returns>The translated and filtered filesystem path used to identify the file resource</returns> + public abstract string TranslateResourcePath(string requestPath); + /// <summary> + /// <para> + /// When an error occurs and is handled by the library, this event is invoked + /// </para> + /// <para> + /// NOTE: This function must be thread-safe! + /// </para> + /// </summary> + /// <param name="errorCode">The error code that was created during processing</param> + /// <param name="entity">The active IHttpEvent representing the faulted request</param> + /// <returns>A value indicating if the entity was proccsed by this call</returns> + public abstract bool ErrorHandler(HttpStatusCode errorCode, IHttpEvent entity); + /// <summary> + /// For pre-processing a request entity before all endpoint lookups are performed + /// </summary> + /// <param name="entity">The http entity to process</param> + /// <returns>The results to return to the file processor, or null of the entity requires further processing</returns> + public abstract ValueTask<FileProcessArgs> PreProcessEntityAsync(HttpEntity entity); + /// <summary> + /// Allows for post processing of a selected <see cref="FileProcessArgs"/> for the given entity + /// </summary> + /// <param name="entity">The http entity to process</param> + /// <param name="chosenRoutine">The selected file processing routine for the given request</param> + public abstract void PostProcessFile(HttpEntity entity, in FileProcessArgs chosenRoutine); + + #region redirects + ///<inheritdoc/> + public IReadOnlyDictionary<string, Redirect> Redirects => _redirects; + + private Dictionary<string, Redirect> _redirects = new(); + + /// <summary> + /// Initializes 301 redirects table from a collection of redirects + /// </summary> + /// <param name="redirs">A collection of redirects</param> + public void SetRedirects(IEnumerable<Redirect> redirs) + { + //To dictionary + Dictionary<string, Redirect> r = redirs.ToDictionary(r => r.Url, r => r, StringComparer.OrdinalIgnoreCase); + //Swap + _ = Interlocked.Exchange(ref _redirects, r); + } + + #endregion + + #region sessions + + /// <summary> + /// An <see cref="ISessionProvider"/> that connects stateful sessions to + /// HTTP connections + /// </summary> + private ISessionProvider? Sessions; + + /// <summary> + /// Sets or resets the current <see cref="ISessionProvider"/> + /// for all connections + /// </summary> + /// <param name="sp">The new <see cref="ISessionProvider"/></param> + public void SetSessionProvider(ISessionProvider? sp) => _ = Interlocked.Exchange(ref Sessions, sp); + + #endregion + + #region router + + /// <summary> + /// An <see cref="IPageRouter"/> to route files to be processed + /// </summary> + private IPageRouter? Router; + + /// <summary> + /// Sets or resets the current <see cref="IPageRouter"/> + /// for all connections + /// </summary> + /// <param name="router"><see cref="IPageRouter"/> to route incomming connections</param> + public void SetPageRouter(IPageRouter? router) => _ = Interlocked.Exchange(ref Router, router); + + #endregion + + #region Virtual Endpoints + + /* + * Wrapper class for converting IHttpEvent endpoints to + * httpEntityEndpoints + */ + private class EvEndpointWrapper : IVirtualEndpoint<HttpEntity> + { + private readonly IVirtualEndpoint<IHttpEvent> _wrapped; + public EvEndpointWrapper(IVirtualEndpoint<IHttpEvent> wrapping) => _wrapped = wrapping; + string IEndpoint.Path => _wrapped.Path; + ValueTask<VfReturnType> IVirtualEndpoint<HttpEntity>.Process(HttpEntity entity) => _wrapped.Process(entity); + } + + + /* + * The VE table is read-only for the processor and my only + * be updated by the application via the methods below + * + * Since it would be very inefficient to track endpoint users + * using locks, we can assume any endpoint that is currently + * processing requests cannot be stopped, so we just focus on + * swapping the table when updates need to be made. + * + * This means calls to modify the table will read the table + * (clone it), modify the local copy, then exhange it for + * the active table so new requests will be processed on the + * new table. + * + * To make the calls to modify the table thread safe, a lock is + * held while modification operations run, then the updated + * copy is published. Any threads reading the old table + * will continue to use a stale endpoint. + */ + + /// <summary> + /// A "lookup table" that represents virtual endpoints to be processed when an + /// incomming connection matches its path parameter + /// </summary> + private Dictionary<string, IVirtualEndpoint<HttpEntity>> VirtualEndpoints = new(); + + + /* + * A lock that is held by callers that intend to + * modify the vep table at the same time + */ + private readonly object VeUpdateLock = new(); + + + /// <summary> + /// Determines the endpoint type(s) and adds them to the endpoint store(s) as necessary + /// </summary> + /// <param name="endpoints">Params array of endpoints to add to the store</param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + public void AddEndpoint(params IEndpoint[] endpoints) + { + //Check + _ = endpoints ?? throw new ArgumentNullException(nameof(endpoints)); + //Make sure all endpoints specify a path + if(endpoints.Any(static e => string.IsNullOrWhiteSpace(e?.Path))) + { + throw new ArgumentException("Endpoints array contains one or more empty endpoints"); + } + + if (endpoints.Length == 0) + { + return; + } + + //Get virtual endpoints + IEnumerable<IVirtualEndpoint<HttpEntity>> eps = endpoints + .Where(static e => e is IVirtualEndpoint<HttpEntity>) + .Select(static e => (IVirtualEndpoint<HttpEntity>)e); + + //Get http event endpoints and create wrapper classes for conversion + IEnumerable<IVirtualEndpoint<HttpEntity>> evs = endpoints + .Where(static e => e is IVirtualEndpoint<IHttpEvent>) + .Select(static e => new EvEndpointWrapper((e as IVirtualEndpoint<IHttpEvent>)!)); + + //Uinion endpoints by their paths to combine them + IEnumerable<IVirtualEndpoint<HttpEntity>> allEndpoints = eps.UnionBy(evs, static s => s.Path); + + lock (VeUpdateLock) + { + //Clone the current dictonary + Dictionary<string, IVirtualEndpoint<HttpEntity>> newTable = new(VirtualEndpoints, StringComparer.OrdinalIgnoreCase); + //Insert the new eps, and/or overwrite old eps + foreach(IVirtualEndpoint<HttpEntity> ep in allEndpoints) + { + newTable.Add(ep.Path, ep); + } + + //Store the new table + _ = Interlocked.Exchange(ref VirtualEndpoints, newTable); + } + } + + /// <summary> + /// Removes the specified endpoint from the virtual store and oauthendpoints if eneabled and found + /// </summary> + /// <param name="eps">A collection of endpoints to remove from the table</param> + public void RemoveEndpoint(params IEndpoint[] eps) + { + _ = eps ?? throw new ArgumentNullException(nameof(eps)); + //Call remove on path + RemoveVirtualEndpoint(eps.Select(static s => s.Path).ToArray()); + } + + /// <summary> + /// Stops listening for connections to the specified <see cref="IVirtualEndpoint{T}"/> identified by its path + /// </summary> + /// <param name="paths">An array of endpoint paths to remove from the table</param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public void RemoveVirtualEndpoint(params string[] paths) + { + _ = paths ?? throw new ArgumentNullException(nameof(paths)); + //Make sure all endpoints specify a path + if (paths.Any(static e => string.IsNullOrWhiteSpace(e))) + { + throw new ArgumentException("Paths array contains one or more empty strings"); + } + + if(paths.Length == 0) + { + return; + } + + //take update lock + lock (VeUpdateLock) + { + //Clone the current dictonary + Dictionary<string, IVirtualEndpoint<HttpEntity>> newTable = new(VirtualEndpoints, StringComparer.OrdinalIgnoreCase); + + foreach(string eps in paths) + { + _ = newTable.Remove(eps); + } + //Store the new table ony if the endpoint existed + _ = Interlocked.Exchange(ref VirtualEndpoints, newTable); + } + } + + #endregion + + ///<inheritdoc/> + public virtual async ValueTask ClientConnectedAsync(IHttpEvent httpEvent) + { + //load ref to session provider + ISessionProvider? _sessions = Sessions; + + //Set ambient processor context + _currentProcessor.Value = this; + //Start cancellation token + CancellationTokenSource timeout = new(Options.ExecutionTimeout); + try + { + //Session handle, default to the shared empty session + SessionHandle sessionHandle = default; + + //If sessions are set, get a session for the current connection + if (_sessions != null) + { + //Get the session + sessionHandle = await _sessions.GetSessionAsync(httpEvent, timeout.Token); + //If the processor had an error recovering the session, return the result to the processor + if (sessionHandle.EntityStatus != FileProcessArgs.Continue) + { + ProcessFile(httpEvent, sessionHandle.EntityStatus); + return; + } + } + try + { + //Setup entity + HttpEntity entity = new(httpEvent, this, in sessionHandle, timeout.Token); + //Pre-process entity + FileProcessArgs preProc = await PreProcessEntityAsync(entity); + //If preprocess returned a value, exit + if (preProc != FileProcessArgs.Continue) + { + ProcessFile(httpEvent, in preProc); + return; + } + + if (VirtualEndpoints.Count > 0) + { + //Process a virtual file + FileProcessArgs virtualArgs = await ProcessVirtualAsync(entity); + //If the entity was processed, exit + if (virtualArgs != FileProcessArgs.Continue) + { + ProcessFile(httpEvent, in virtualArgs); + return; + } + } + //If no virtual processor handled the ws request, deny it + if (entity.Server.IsWebSocketRequest) + { + ProcessFile(httpEvent, in FileProcessArgs.Deny); + return; + } + //Finally process as file + FileProcessArgs args = await RouteFileAsync(entity); + //Finally process the file + ProcessFile(httpEvent, in args); + } + finally + { + //Capture all session release exceptions + try + { + //Release the session + await sessionHandle.ReleaseAsync(httpEvent); + } + catch (Exception ex) + { + Log.Error(ex, "Exception raised while releasing the assocated session"); + } + } + } + catch (ContentTypeUnacceptableException) + { + /* + * The user application attempted to set a content that the client does not accept + * Assuming this exception was uncaught by application code, there should not be + * any response body, either way we should respond with the unacceptable status code + */ + CloseWithError(HttpStatusCode.NotAcceptable, httpEvent); + } + catch (TerminateConnectionException) + { + throw; + } + catch (ResourceUpdateFailedException ruf) + { + Log.Warn(ruf); + CloseWithError(HttpStatusCode.ServiceUnavailable, httpEvent); + } + catch (SessionException se) + { + Log.Warn(se, "An exception was raised while attempting to get or save a session"); + CloseWithError(HttpStatusCode.ServiceUnavailable, httpEvent); + return; + } + catch (OperationCanceledException oce) + { + Log.Warn(oce, "Request execution time exceeded, connection terminated"); + CloseWithError(HttpStatusCode.ServiceUnavailable, httpEvent); + } + catch (IOException ioe) when (ioe.InnerException is SocketException) + { + throw; + } + catch (Exception ex) + { + Log.Warn(ex, "Unhandled exception during application code execution."); + //Invoke the root error handler + CloseWithError(HttpStatusCode.InternalServerError, httpEvent); + } + finally + { + timeout.Dispose(); + _currentProcessor.Value = null; + } + } + + /// <summary> + /// Accepts the entity to process a file for an the selected <see cref="FileProcessArgs"/> + /// by user code and determines what file-system file to open and respond to the connection with. + /// </summary> + /// <param name="entity">The entity to process the file for</param> + /// <param name="args">The selected <see cref="FileProcessArgs"/> to determine what file to process</param> + protected virtual void ProcessFile(IHttpEvent entity, in FileProcessArgs args) + { + try + { + string? filename = null; + //Switch on routine + switch (args.Routine) + { + //Close the connection with an error state + case FpRoutine.Error: + CloseWithError(HttpStatusCode.InternalServerError, entity); + return; + //Redirect the user + case FpRoutine.Redirect: + //Set status code + entity.Redirect(RedirectType.Found, args.Alternate); + return; + //Deny + case FpRoutine.Deny: + CloseWithError(HttpStatusCode.Forbidden, entity); + return; + //Not return not found + case FpRoutine.NotFound: + CloseWithError(HttpStatusCode.NotFound, entity); + return; + //Serve other file + case FpRoutine.ServeOther: + { + //Use the specified relative alternate path the user specified + if (FindResourceInRoot(args.Alternate, out string otherPath)) + { + filename = otherPath; + } + } + break; + //Normal file lookup + case FpRoutine.Continue: + { + //Lookup the file based on the client requested local path + if (FindResourceInRoot(entity.Server.Path, out string path)) + { + filename = path; + } + } + break; + //The user indicated that the file is a fully qualified path, and should be treated directly + case FpRoutine.ServeOtherFQ: + { + //Get the absolute path of the file rooted in the current server root and determine if it exists + if (FindResourceInRoot(args.Alternate, true, out string fqPath)) + { + filename = fqPath; + } + } + break; + //The user has indicated they handled all necessary action, and we will exit + case FpRoutine.VirtualSkip: + return; + default: + break; + } + //If the file was not set or the request method is not a GET (or HEAD), return not-found + if (filename == null || (entity.Server.Method & (HttpMethod.GET | HttpMethod.HEAD)) == 0) + { + CloseWithError(HttpStatusCode.NotFound, entity); + return; + } + DateTime fileLastModified = File.GetLastWriteTimeUtc(filename); + //See if the last modifed header was set + DateTimeOffset? ifModifedSince = entity.Server.LastModified(); + //If the header was set, check the date, if the file has been modified since, continue sending the file + if (ifModifedSince != null && ifModifedSince.Value > fileLastModified) + { + //File has not been modified + entity.CloseResponse(HttpStatusCode.NotModified); + return; + } + //Get the content type of he file + ContentType fileType = HttpHelpers.GetContentTypeFromFile(filename); + //Make sure the client accepts the content type + if (entity.Server.Accepts(fileType)) + { + //set last modified time as the files last write time + entity.Server.LastModified(fileLastModified); + //try to open the selected file for reading and allow sharing + FileStream fs = new (filename, FileMode.Open, FileAccess.Read, FileShare.Read); + //Check for range + if (entity.Server.Range != null && entity.Server.Range.Item1 > 0) + { + //Seek the stream to the specified position + fs.Position = entity.Server.Range.Item1; + entity.CloseResponse(HttpStatusCode.PartialContent, fileType, fs); + } + else + { + //send the whole file + entity.CloseResponse(HttpStatusCode.OK, fileType, fs); + } + return; + } + else + { + //Unacceptable + CloseWithError(HttpStatusCode.NotAcceptable, entity); + return; + } + } + catch (IOException ioe) + { + Log.Information(ioe, "Unhandled exception during file opening."); + CloseWithError(HttpStatusCode.Locked, entity); + return; + } + catch (Exception ex) + { + Log.Information(ex, "Unhandled exception during file opening."); + //Invoke the root error handler + CloseWithError(HttpStatusCode.InternalServerError, entity); + return; + } + } + + private void CloseWithError(HttpStatusCode code, IHttpEvent entity) + { + //Invoke the inherited error handler + if (!ErrorHandler(code, entity)) + { + //Disable cache + entity.Server.SetNoCache(); + //Error handler does not have a response for the error code, so return a generic error code + entity.CloseResponse(code); + } + } + + + /// <summary> + /// If virtual endpoints are enabled, checks for the existance of an + /// endpoint and attmepts to process that endpoint. + /// </summary> + /// <param name="entity">The http entity to proccess</param> + /// <returns>The results to return to the file processor, or null of the entity requires further processing</returns> + protected virtual async ValueTask<FileProcessArgs> ProcessVirtualAsync(HttpEntity entity) + { + //See if the virtual file is servicable + if (!VirtualEndpoints.TryGetValue(entity.Server.Path, out IVirtualEndpoint<HttpEntity>? vf)) + { + return FileProcessArgs.Continue; + } + + //Invoke the page handler process method + VfReturnType rt = await vf.Process(entity); + + if (rt == VfReturnType.VirtualSkip) + { + //Virtual file was handled by the handler + return FileProcessArgs.VirtualSkip; + } + else if(rt == VfReturnType.ProcessAsFile) + { + return FileProcessArgs.Continue; + } + + //If not a get request, process it directly + if (entity.Server.Method == HttpMethod.GET) + { + switch (rt) + { + case VfReturnType.Forbidden: + return FileProcessArgs.Deny; + case VfReturnType.NotFound: + return FileProcessArgs.NotFound; + case VfReturnType.Error: + return FileProcessArgs.Error; + default: + break; + } + } + + switch (rt) + { + case VfReturnType.Forbidden: + entity.CloseResponse(HttpStatusCode.Forbidden); + break; + case VfReturnType.BadRequest: + entity.CloseResponse(HttpStatusCode.BadRequest); + break; + case VfReturnType.Error: + entity.CloseResponse(HttpStatusCode.InternalServerError); + break; + case VfReturnType.NotFound: + default: + entity.CloseResponse(HttpStatusCode.NotFound); + break; + } + + return FileProcessArgs.VirtualSkip; + } + + /// <summary> + /// Determines the best <see cref="FileProcessArgs"/> processing response for the given connection. + /// Alternativley may respond to the entity directly. + /// </summary> + /// <param name="entity">The http entity to process</param> + /// <returns>The results to return to the file processor, this method must return an argument</returns> + protected virtual async ValueTask<FileProcessArgs> RouteFileAsync(HttpEntity entity) + { + //Read local copy of the router + + IPageRouter? router = Router; + //Make sure router is set + if (router == null) + { + return FileProcessArgs.Continue; + } + //Get a file routine + FileProcessArgs routine = await router.RouteAsync(entity); + //Call post processor method + PostProcessFile(entity, in routine); + //Return the routine + return routine; + } + + + /// <summary> + /// Finds the file specified by the request and the server root the user has requested. + /// Determines if it exists, has permissions to access it, and allowed file attributes. + /// Also finds default files and files without extensions + /// </summary> + public bool FindResourceInRoot(string resourcePath, bool fullyQualified, out string path) + { + //Special case where user's can specify a fullly qualified path (meant to reach a remote file, eg UNC/network share or other disk) + if (fullyQualified && Path.IsPathRooted(resourcePath) && Path.IsPathFullyQualified(resourcePath) && FileOperations.FileExists(resourcePath)) + { + path = resourcePath; + return true; + } + //Otherwise invoke non fully qualified path + return FindResourceInRoot(resourcePath, out path); + } + + /// <summary> + /// Determines if a requested resource exists within the <see cref="EventProcessor"/> and is allowed to be accessed. + /// </summary> + /// <param name="resourcePath">The path to the resource</param> + /// <param name="path">An out parameter that is set to the absolute path to the existing and accessable resource</param> + /// <returns>True if the resource exists and is allowed to be accessed</returns> + public bool FindResourceInRoot(string resourcePath, out string path) + { + //Check after fully qualified path name because above is a special case + path = TranslateResourcePath(resourcePath); + string extension = Path.GetExtension(path); + //Make sure extension isnt blocked + if (Options.ExcludedExtensions.Contains(extension)) + { + return false; + } + //Trailing / means dir, so look for a default file (index.html etc) (most likely so check first?) + if (Path.EndsInDirectorySeparator(path)) + { + string comp = path; + //Find default file if blank + foreach (string d in Options.DefaultFiles) + { + path = Path.Combine(comp, d); + if (FileOperations.FileExists(path)) + { + //Get attributes + FileAttributes att = FileOperations.GetAttributes(path); + //Make sure the file is accessable and isnt an unsafe file + return ((att & Options.AllowedAttributes) > 0) && ((att & Options.DissallowedAttributes) == 0); + } + } + } + //try the file as is + else if (FileOperations.FileExists(path)) + { + //Get attributes + FileAttributes att = FileOperations.GetAttributes(path); + //Make sure the file is accessable and isnt an unsafe file + return ((att & Options.AllowedAttributes) > 0) && ((att & Options.DissallowedAttributes) == 0); + } + return false; + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/CollectionsExtensions.cs b/Plugins.Essentials/src/Extensions/CollectionsExtensions.cs new file mode 100644 index 0000000..9500d5e --- /dev/null +++ b/Plugins.Essentials/src/Extensions/CollectionsExtensions.cs @@ -0,0 +1,85 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: CollectionsExtensions.cs +* +* CollectionsExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + /// <summary> + /// + /// </summary> + public static class CollectionsExtensions + { + /// <summary> + /// Gets a value by the specified key if it exsits and the value is not null/empty + /// </summary> + /// <param name="dict"></param> + /// <param name="key">Key associated with the value</param> + /// <param name="value">Value associated with the key</param> + /// <returns>True of the key is found and is not noll/empty, false otherwise</returns> + public static bool TryGetNonEmptyValue(this IReadOnlyDictionary<string, string> dict, string key, [MaybeNullWhen(false)] out string value) + { + if (dict.TryGetValue(key, out string? val) && !string.IsNullOrWhiteSpace(val)) + { + value = val; + return true; + } + value = null; + return false; + } + /// <summary> + /// Determines if an argument was set in a <see cref="IReadOnlyDictionary{TKey, TValue}"/> by comparing + /// the value stored at the key, to the type argument + /// </summary> + /// <param name="dict"></param> + /// <param name="key">The argument's key</param> + /// <param name="argument">The argument to compare against</param> + /// <returns> + /// True if the key was found, and the value at the key is equal to the type parameter. False if the key is null/empty, or the + /// value does not match the specified type + /// </returns> + /// <exception cref="ArgumentNullException"></exception> + public static bool IsArgumentSet(this IReadOnlyDictionary<string, string> dict, string key, ReadOnlySpan<char> argument) + { + //Try to get the value from the dict, if the value is null casting it to span (implicitly) should stop null excpetions and return false + return dict.TryGetValue(key, out string? value) && string.GetHashCode(argument) == string.GetHashCode(value); + } + /// <summary> + /// + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="TValue"></typeparam> + /// <param name="dict"></param> + /// <param name="key"></param> + /// <returns></returns> + public static TValue? GetValueOrDefault<TKey, TValue>(this IDictionary<TKey, TValue> dict, TKey key) + { + return dict.TryGetValue(key, out TValue? value) ? value : default; + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/ConnectionInfoExtensions.cs b/Plugins.Essentials/src/Extensions/ConnectionInfoExtensions.cs new file mode 100644 index 0000000..ba01132 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/ConnectionInfoExtensions.cs @@ -0,0 +1,361 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ConnectionInfoExtensions.cs +* +* ConnectionInfoExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Diagnostics.CodeAnalysis; +using System.Net; +using System.Runtime.CompilerServices; + +using VNLib.Net.Http; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + /// <summary> + /// Provides <see cref="ConnectionInfo"/> extension methods + /// for common use cases + /// </summary> + public static class IConnectionInfoExtensions + { + public const string SEC_HEADER_MODE = "Sec-Fetch-Mode"; + public const string SEC_HEADER_SITE = "Sec-Fetch-Site"; + public const string SEC_HEADER_USER = "Sec-Fetch-User"; + public const string SEC_HEADER_DEST = "Sec-Fetch-Dest"; + public const string X_FORWARDED_FOR_HEADER = "x-forwarded-for"; + public const string X_FORWARDED_PROTO_HEADER = "x-forwarded-proto"; + public const string DNT_HEADER = "dnt"; + + /// <summary> + /// Cache-Control header value for disabling cache + /// </summary> + public static readonly string NO_CACHE_RESPONSE_HEADER_VALUE = HttpHelpers.GetCacheString(CacheType.NoCache | CacheType.NoStore | CacheType.Revalidate); + + /// <summary> + /// Gets the <see cref="HttpRequestHeader.IfModifiedSince"/> header value and converts its value to a datetime value + /// </summary> + /// <returns>The if modified-since header date-time, null if the header was not set or the value was invalid</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static DateTimeOffset? LastModified(this IConnectionInfo server) + { + //Get the if-modified-since header + string? ifModifiedSince = server.Headers[HttpRequestHeader.IfModifiedSince]; + //Make sure tis set and try to convert it to a date-time structure + return DateTimeOffset.TryParse(ifModifiedSince, out DateTimeOffset d) ? d : null; + } + + /// <summary> + /// Sets the last-modified response header value + /// </summary> + /// <param name="server"></param> + /// <param name="value">Time the entity was last modified</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void LastModified(this IConnectionInfo server, DateTimeOffset value) + { + server.Headers[HttpResponseHeader.LastModified] = value.ToString("R"); + } + + /// <summary> + /// Is the connection requesting cors + /// </summary> + /// <returns>true if the user-agent specified the cors security header</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsCors(this IConnectionInfo server) => "cors".Equals(server.Headers[SEC_HEADER_MODE], StringComparison.OrdinalIgnoreCase); + /// <summary> + /// Determines if the User-Agent specified "cross-site" in the Sec-Site header, OR + /// the connection spcified an origin header and the origin's host does not match the + /// requested host + /// </summary> + /// <returns>true if the request originated from a site other than the current one</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsCrossSite(this IConnectionInfo server) + { + return "cross-site".Equals(server.Headers[SEC_HEADER_SITE], StringComparison.OrdinalIgnoreCase) + || (server.Origin != null && !server.RequestUri.DnsSafeHost.Equals(server.Origin.DnsSafeHost, StringComparison.Ordinal)); + } + /// <summary> + /// Is the connection user-agent created, or automatic + /// </summary> + /// <param name="server"></param> + /// <returns>true if sec-user header was set to "?1"</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsUserInvoked(this IConnectionInfo server) => "?1".Equals(server.Headers[SEC_HEADER_USER], StringComparison.OrdinalIgnoreCase); + /// <summary> + /// Was this request created from normal user navigation + /// </summary> + /// <returns>true if sec-mode set to "navigate"</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsNavigation(this IConnectionInfo server) => "navigate".Equals(server.Headers[SEC_HEADER_MODE], StringComparison.OrdinalIgnoreCase); + /// <summary> + /// Determines if the client specified "no-cache" for the cache control header, signalling they do not wish to cache the entity + /// </summary> + /// <returns>True if <see cref="HttpRequestHeader.CacheControl"/> contains the string "no-cache", false otherwise</returns> + public static bool NoCache(this IConnectionInfo server) + { + string? cache_header = server.Headers[HttpRequestHeader.CacheControl]; + return !string.IsNullOrWhiteSpace(cache_header) && cache_header.Contains("no-cache", StringComparison.OrdinalIgnoreCase); + } + /// <summary> + /// Sets the response cache headers to match the requested caching type. Does not check against request headers + /// </summary> + /// <param name="server"></param> + /// <param name="type">One or more <see cref="CacheType"/> flags that identify the way the entity can be cached</param> + /// <param name="maxAge">The max age the entity is valid for</param> + public static void SetCache(this IConnectionInfo server, CacheType type, TimeSpan maxAge) + { + //If no cache flag is set, set the pragma header to no-cache + if((type & CacheType.NoCache) > 0) + { + server.Headers[HttpResponseHeader.Pragma] = "no-cache"; + } + //Set the cache hader string using the http helper class + server.Headers[HttpResponseHeader.CacheControl] = HttpHelpers.GetCacheString(type, maxAge); + } + /// <summary> + /// Sets the Cache-Control response header to <see cref="NO_CACHE_RESPONSE_HEADER_VALUE"/> + /// and the pragma response header to 'no-cache' + /// </summary> + /// <param name="server"></param> + public static void SetNoCache(this IConnectionInfo server) + { + //Set default nocache string + server.Headers[HttpResponseHeader.CacheControl] = NO_CACHE_RESPONSE_HEADER_VALUE; + server.Headers[HttpResponseHeader.Pragma] = "no-cache"; + } + + /// <summary> + /// Gets a value indicating whether the port number in the request is equivalent to the port number + /// on the local server. + /// </summary> + /// <returns>True if the port number in the <see cref="ConnectionInfo.RequestUri"/> matches the + /// <see cref="ConnectionInfo.LocalEndpoint"/> port false if they do not match + /// </returns> + /// <remarks> + /// Users should call this method to help prevent port based attacks if your + /// code relies on the port number of the <see cref="ConnectionInfo.RequestUri"/> + /// </remarks> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool EnpointPortsMatch(this IConnectionInfo server) + { + return server.RequestUri.Port == server.LocalEndpoint.Port; + } + /// <summary> + /// Determines if the host of the current request URI matches the referer header host + /// </summary> + /// <returns>True if the request host and the referer host paremeters match, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool RefererMatch(this IConnectionInfo server) + { + return server.RequestUri.DnsSafeHost.Equals(server.Referer?.DnsSafeHost, StringComparison.OrdinalIgnoreCase); + } + /// <summary> + /// Expires a client's cookie + /// </summary> + /// <param name="server"></param> + /// <param name="name"></param> + /// <param name="domain"></param> + /// <param name="path"></param> + /// <param name="sameSite"></param> + /// <param name="secure"></param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ExpireCookie(this IConnectionInfo server, string name, string domain = "", string path = "/", CookieSameSite sameSite = CookieSameSite.None, bool secure = false) + { + server.SetCookie(name, string.Empty, domain, path, TimeSpan.Zero, sameSite, false, secure); + } + /// <summary> + /// Sets a cookie with an infinite (session life-span) + /// </summary> + /// <param name="server"></param> + /// <param name="name"></param> + /// <param name="value"></param> + /// <param name="domain"></param> + /// <param name="path"></param> + /// <param name="sameSite"></param> + /// <param name="httpOnly"></param> + /// <param name="secure"></param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetSessionCookie( + this IConnectionInfo server, + string name, + string value, + string domain = "", + string path = "/", + CookieSameSite sameSite = CookieSameSite.None, + bool httpOnly = false, + bool secure = false) + { + server.SetCookie(name, value, domain, path, TimeSpan.MaxValue, sameSite, httpOnly, secure); + } + + /// <summary> + /// Sets a cookie with an infinite (session life-span) + /// </summary> + /// <param name="server"></param> + /// <param name="name"></param> + /// <param name="value"></param> + /// <param name="domain"></param> + /// <param name="path"></param> + /// <param name="sameSite"></param> + /// <param name="httpOnly"></param> + /// <param name="secure"></param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetCookie( + this IConnectionInfo server, + string name, + string value, + TimeSpan expires, + string domain = "", + string path = "/", + CookieSameSite sameSite = CookieSameSite.None, + bool httpOnly = false, + bool secure = false) + { + server.SetCookie(name, value, domain, path, expires, sameSite, httpOnly, secure); + } + + /// <summary> + /// Is the current connection a "browser" ? + /// </summary> + /// <param name="server"></param> + /// <returns>true if the user agent string contains "Mozilla" and does not contain "bot", false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsBrowser(this IConnectionInfo server) + { + //Get user-agent and determine if its a browser + return server.UserAgent != null && !server.UserAgent.Contains("bot", StringComparison.OrdinalIgnoreCase) && server.UserAgent.Contains("Mozilla", StringComparison.OrdinalIgnoreCase); + } + /// <summary> + /// Determines if the current connection is the loopback/internal network adapter + /// </summary> + /// <param name="server"></param> + /// <returns>True of the connection was made from the local machine</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsLoopBack(this IConnectionInfo server) + { + IPAddress realIp = server.GetTrustedIp(); + return IPAddress.Any.Equals(realIp) || IPAddress.Loopback.Equals(realIp); + } + + /// <summary> + /// Did the connection set the dnt header? + /// </summary> + /// <returns>true if the connection specified the dnt header, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool DNT(this IConnectionInfo server) => !string.IsNullOrWhiteSpace(server.Headers[DNT_HEADER]); + + /// <summary> + /// Determins if the current connection is behind a trusted downstream server + /// </summary> + /// <param name="server"></param> + /// <returns>True if the connection came from a trusted downstream server, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsBehindDownStreamServer(this IConnectionInfo server) + { + //See if there is an ambient event processor + EventProcessor? ev = EventProcessor.Current; + //See if the connection is coming from an downstream server + return ev != null && ev.Options.DownStreamServers.Contains(server.RemoteEndpoint.Address); + } + + /// <summary> + /// Gets the real IP address of the request if behind a trusted downstream server, otherwise returns the transport remote ip address + /// </summary> + /// <param name="server"></param> + /// <returns>The real ip of the connection</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static IPAddress GetTrustedIp(this IConnectionInfo server) => GetTrustedIp(server, server.IsBehindDownStreamServer()); + /// <summary> + /// Gets the real IP address of the request if behind a trusted downstream server, otherwise returns the transport remote ip address + /// </summary> + /// <param name="server"></param> + /// <param name="isTrusted"></param> + /// <returns>The real ip of the connection</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal static IPAddress GetTrustedIp(this IConnectionInfo server, bool isTrusted) + { + //If the connection is not trusted, then ignore header parsing + if (isTrusted) + { + //Nginx sets a header identifying the remote ip address so parse it + string? real_ip = server.Headers[X_FORWARDED_FOR_HEADER]; + //If the real-ip header is set, try to parse is and return the address found, otherwise return the remote ep + return !string.IsNullOrWhiteSpace(real_ip) && IPAddress.TryParse(real_ip, out IPAddress? addr) ? addr : server.RemoteEndpoint.Address; + } + else + { + return server.RemoteEndpoint.Address; + } + } + + /// <summary> + /// Gets a value that determines if the connection is using tls, locally + /// or behind a trusted downstream server that is using tls. + /// </summary> + /// <param name="server"></param> + /// <returns>True if the connection is secure, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsSecure(this IConnectionInfo server) + { + //Get value of the trusted downstream server + return IsSecure(server, server.IsBehindDownStreamServer()); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + internal static bool IsSecure(this IConnectionInfo server, bool isTrusted) + { + //If the connection is not trusted, then ignore header parsing + if (isTrusted) + { + //Standard https protocol header + string? protocol = server.Headers[X_FORWARDED_PROTO_HEADER]; + //If the header is set and equals https then tls is being used + return string.IsNullOrWhiteSpace(protocol) ? server.IsSecure : "https".Equals(protocol, StringComparison.OrdinalIgnoreCase); + } + else + { + return server.IsSecure; + } + } + + /// <summary> + /// Was the connection made on a local network to the server? NOTE: Use with caution + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsLocalConnection(this IConnectionInfo server) => server.LocalEndpoint.Address.IsLocalSubnet(server.GetTrustedIp()); + + /// <summary> + /// Get a cookie from the current request + /// </summary> + /// <param name="server"></param> + /// <param name="name">Name/ID of cookie</param> + /// <param name="cookieValue">Is set to cookie if found, or null if not</param> + /// <returns>True if cookie exists and was retrieved</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool GetCookie(this IConnectionInfo server, string name, [NotNullWhen(true)] out string? cookieValue) + { + //Try to get a cookie from the request + return server.RequestCookies.TryGetValue(name, out cookieValue); + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/EssentialHttpEventExtensions.cs b/Plugins.Essentials/src/Extensions/EssentialHttpEventExtensions.cs new file mode 100644 index 0000000..9458487 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/EssentialHttpEventExtensions.cs @@ -0,0 +1,848 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: EssentialHttpEventExtensions.cs +* +* EssentialHttpEventExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; + +using VNLib.Net.Http; +using VNLib.Hashing; +using VNLib.Utils; +using VNLib.Utils.Extensions; +using VNLib.Utils.Memory.Caching; +using static VNLib.Plugins.Essentials.Statics; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + + /// <summary> + /// Provides extension methods for manipulating <see cref="HttpEvent"/>s + /// </summary> + public static class EssentialHttpEventExtensions + { + public const string BEARER_STRING = "Bearer"; + private static readonly int BEARER_LEN = BEARER_STRING.Length; + + /* + * Pooled/tlocal serializers + */ + private static ThreadLocal<Utf8JsonWriter> LocalSerializer { get; } = new(() => new(Stream.Null)); + private static IObjectRental<JsonResponse> ResponsePool { get; } = ObjectRental.Create(ResponseCtor); + private static JsonResponse ResponseCtor() => new(ResponsePool); + + #region Response Configuring + + /// <summary> + /// Attempts to serialize the JSON object (with default SR_OPTIONS) to binary and configure the response for a JSON message body + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="ev"></param> + /// <param name="code">The <see cref="HttpStatusCode"/> result of the connection</param> + /// <param name="response">The JSON object to serialzie and send as response body</param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseJson<T>(this IHttpEvent ev, HttpStatusCode code, T response) => CloseResponseJson(ev, code, response, SR_OPTIONS); + + /// <summary> + /// Attempts to serialize the JSON object to binary and configure the response for a JSON message body + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="ev"></param> + /// <param name="code">The <see cref="HttpStatusCode"/> result of the connection</param> + /// <param name="response">The JSON object to serialzie and send as response body</param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during serialization</param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseJson<T>(this IHttpEvent ev, HttpStatusCode code, T response, JsonSerializerOptions? options) + { + JsonResponse rbuf = ResponsePool.Rent(); + try + { + //Serialze the object on the thread local serializer + LocalSerializer.Value!.Serialize(rbuf, response, options); + + //Set the response as the buffer, + ev.CloseResponse(code, ContentType.Json, rbuf); + } + catch + { + //Return back to pool on error + ResponsePool.Return(rbuf); + throw; + } + } + + /// <summary> + /// Attempts to serialize the JSON object to binary and configure the response for a JSON message body + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The <see cref="HttpStatusCode"/> result of the connection</param> + /// <param name="response">The JSON object to serialzie and send as response body</param> + /// <param name="type">The type to use during de-serialization</param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseJson(this IHttpEvent ev, HttpStatusCode code, object response, Type type) => CloseResponseJson(ev, code, response, type, SR_OPTIONS); + + /// <summary> + /// Attempts to serialize the JSON object to binary and configure the response for a JSON message body + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The <see cref="HttpStatusCode"/> result of the connection</param> + /// <param name="response">The JSON object to serialzie and send as response body</param> + /// <param name="type">The type to use during de-serialization</param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during serialization</param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseJson(this IHttpEvent ev, HttpStatusCode code, object response, Type type, JsonSerializerOptions? options) + { + JsonResponse rbuf = ResponsePool.Rent(); + try + { + //Serialze the object on the thread local serializer + LocalSerializer.Value!.Serialize(rbuf, response, type, options); + + //Set the response as the buffer, + ev.CloseResponse(code, ContentType.Json, rbuf); + } + catch + { + //Return back to pool on error + ResponsePool.Return(rbuf); + throw; + } + } + + /// <summary> + /// Writes the <see cref="JsonDocument"/> data to a temporary buffer and sets it as the response + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The <see cref="HttpStatusCode"/> result of the connection</param> + /// <param name="data">The <see cref="JsonDocument"/> data to send to client</param> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseJson(this IHttpEvent ev, HttpStatusCode code, JsonDocument data) + { + if(data == null) + { + ev.CloseResponse(code); + return; + } + + JsonResponse rbuf = ResponsePool.Rent(); + try + { + //Serialze the object on the thread local serializer + LocalSerializer.Value!.Serialize(rbuf, data); + + //Set the response as the buffer, + ev.CloseResponse(code, ContentType.Json, rbuf); + } + catch + { + //Return back to pool on error + ResponsePool.Return(rbuf); + throw; + } + } + + /// <summary> + /// Close as response to a client with an <see cref="HttpStatusCode.OK"/> and serializes a <see cref="WebMessage"/> as the message response + /// </summary> + /// <param name="ev"></param> + /// <param name="webm">The <see cref="WebMessage"/> to serialize and response to client with</param> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponse<T>(this IHttpEvent ev, T webm) where T:WebMessage + { + if (webm == null) + { + ev.CloseResponse(HttpStatusCode.OK); + } + else + { + //Respond with json data + ev.CloseResponseJson(HttpStatusCode.OK, webm); + } + } + + /// <summary> + /// Close a response to a connection with a file as an attachment (set content dispostion) + /// </summary> + /// <param name="ev"></param> + /// <param name="code">Status code</param> + /// <param name="file">The <see cref="FileInfo"/> of the desired file to attach</param> + /// <exception cref="IOException"></exception> + /// <exception cref="FileNotFoundException"></exception> + /// <exception cref="UnauthorizedAccessException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + /// <exception cref="System.Security.SecurityException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseAttachment(this IHttpEvent ev, HttpStatusCode code, FileInfo file) + { + //Close with file + ev.CloseResponse(code, file); + //Set content dispostion as attachment (only if successfull) + ev.Server.Headers["Content-Disposition"] = $"attachment; filename=\"{file.Name}\""; + } + + /// <summary> + /// Close a response to a connection with a file as an attachment (set content dispostion) + /// </summary> + /// <param name="ev"></param> + /// <param name="code">Status code</param> + /// <param name="file">The <see cref="FileStream"/> of the desired file to attach</param> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseAttachment(this IHttpEvent ev, HttpStatusCode code, FileStream file) + { + //Close with file + ev.CloseResponse(code, file); + //Set content dispostion as attachment (only if successfull) + ev.Server.Headers["Content-Disposition"] = $"attachment; filename=\"{file.Name}\""; + } + + /// <summary> + /// Close a response to a connection with a file as an attachment (set content dispostion) + /// </summary> + /// <param name="ev"></param> + /// <param name="code">Status code</param> + /// <param name="data">The data to straem to the client as an attatcment</param> + /// <param name="ct">The <see cref="ContentType"/> that represents the file</param> + /// <param name="fileName">The name of the file to attach</param> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponseAttachment(this IHttpEvent ev, HttpStatusCode code, ContentType ct, Stream data, string fileName) + { + //Close with file + ev.CloseResponse(code, ct, data); + //Set content dispostion as attachment (only if successfull) + ev.Server.Headers["Content-Disposition"] = $"attachment; filename=\"{fileName}\""; + } + + /// <summary> + /// Close a response to a connection with a file as the entire response body (not attachment) + /// </summary> + /// <param name="ev"></param> + /// <param name="code">Status code</param> + /// <param name="file">The <see cref="FileInfo"/> of the desired file to attach</param> + /// <exception cref="IOException"></exception> + /// <exception cref="FileNotFoundException"></exception> + /// <exception cref="UnauthorizedAccessException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + /// <exception cref="System.Security.SecurityException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponse(this IHttpEvent ev, HttpStatusCode code, FileInfo file) + { + //Open filestream for file + FileStream fs = file.Open(FileMode.Open, FileAccess.Read, FileShare.Read); + try + { + //Set the input as a stream + ev.CloseResponse(code, fs); + //Set last modified time only if successfull + ev.Server.Headers[HttpResponseHeader.LastModified] = file.LastWriteTimeUtc.ToString("R"); + } + catch + { + //If their is an exception close the stream and re-throw + fs.Dispose(); + throw; + } + } + + /// <summary> + /// Close a response to a connection with a <see cref="FileStream"/> as the entire response body (not attachment) + /// </summary> + /// <param name="ev"></param> + /// <param name="code">Status code</param> + /// <param name="file">The <see cref="FileStream"/> of the desired file to attach</param> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponse(this IHttpEvent ev, HttpStatusCode code, FileStream file) + { + //Get content type from filename + ContentType ct = HttpHelpers.GetContentTypeFromFile(file.Name); + //Set the input as a stream + ev.CloseResponse(code, ct, file); + } + + /// <summary> + /// Close a response to a connection with a character buffer using the server wide + /// <see cref="ConnectionInfo.Encoding"/> encoding + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The response status code</param> + /// <param name="type">The <see cref="ContentType"/> the data represents</param> + /// <param name="data">The character buffer to send</param> + /// <remarks>This method will store an encoded copy as a memory stream, so be careful with large buffers</remarks> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponse(this IHttpEvent ev, HttpStatusCode code, ContentType type, in ReadOnlySpan<char> data) + { + //Get a memory stream using UTF8 encoding + CloseResponse(ev, code, type, in data, ev.Server.Encoding); + } + + /// <summary> + /// Close a response to a connection with a character buffer using the specified encoding type + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The response status code</param> + /// <param name="type">The <see cref="ContentType"/> the data represents</param> + /// <param name="data">The character buffer to send</param> + /// <param name="encoding">The encoding type to use when converting the buffer</param> + /// <remarks>This method will store an encoded copy as a memory stream, so be careful with large buffers</remarks> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponse(this IHttpEvent ev, HttpStatusCode code, ContentType type, in ReadOnlySpan<char> data, Encoding encoding) + { + if (data.IsEmpty) + { + ev.CloseResponse(code); + return; + } + + //Validate encoding + _ = encoding ?? throw new ArgumentNullException(nameof(encoding)); + + //Get new simple memory response + IMemoryResponseReader reader = new SimpleMemoryResponse(data, encoding); + ev.CloseResponse(code, type, reader); + } + + /// <summary> + /// Close a response to a connection by copying the speciifed binary buffer + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The response status code</param> + /// <param name="type">The <see cref="ContentType"/> the data represents</param> + /// <param name="data">The binary buffer to send</param> + /// <remarks>The data paramter is copied into an internal <see cref="IMemoryResponseReader"/></remarks> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void CloseResponse(this IHttpEvent ev, HttpStatusCode code, ContentType type, ReadOnlySpan<byte> data) + { + if (data.IsEmpty) + { + ev.CloseResponse(code); + return; + } + + //Get new simple memory response + IMemoryResponseReader reader = new SimpleMemoryResponse(data); + ev.CloseResponse(code, type, reader); + } + + /// <summary> + /// Close a response to a connection with a relative file within the current root's directory + /// </summary> + /// <param name="entity"></param> + /// <param name="code">The status code to set the response as</param> + /// <param name="filePath">The path of the relative file to send</param> + /// <returns>True if the file was found, false if the file does not exist or cannot be accessed</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool CloseWithRelativeFile(this HttpEntity entity, HttpStatusCode code, string filePath) + { + //See if file exists and is within the root's directory + if (entity.RequestedRoot.FindResourceInRoot(filePath, out string realPath)) + { + //get file-info + FileInfo realFile = new(realPath); + //Close the response with the file stream + entity.CloseResponse(code, realFile); + return true; + } + return false; + } + + /// <summary> + /// Redirects a client using the specified <see cref="RedirectType"/> + /// </summary> + /// <param name="ev"></param> + /// <param name="type">The <see cref="RedirectType"/> redirection type</param> + /// <param name="location">Location to direct client to, sets the "Location" header</param> + /// <remarks>Sets required headers for redirection, disables cache control, and returns the status code to the client</remarks> + /// <exception cref="UriFormatException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Redirect(this IHttpEvent ev, RedirectType type, string location) + { + Redirect(ev, type, new Uri(location, UriKind.RelativeOrAbsolute)); + } + + /// <summary> + /// Redirects a client using the specified <see cref="RedirectType"/> + /// </summary> + /// <param name="ev"></param> + /// <param name="type">The <see cref="RedirectType"/> redirection type</param> + /// <param name="location">Location to direct client to, sets the "Location" header</param> + /// <remarks>Sets required headers for redirection, disables cache control, and returns the status code to the client</remarks> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Redirect(this IHttpEvent ev, RedirectType type, Uri location) + { + //Encode the string for propery http url formatting and set the location header + ev.Server.Headers[HttpResponseHeader.Location] = location.ToString(); + ev.Server.SetNoCache(); + //Set redirect the ressponse redirect code type + ev.CloseResponse((HttpStatusCode)type); + } + + #endregion + + /// <summary> + /// Attempts to read and deserialize a JSON object from the reqeust body (form data or urlencoded) + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="ev"></param> + /// <param name="key">Request argument key (name)</param> + /// <param name="obj"></param> + /// <returns>true if the argument was found and successfully converted to json</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidJsonRequestException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool TryGetJsonFromArg<T>(this IHttpEvent ev, string key, out T? obj) => TryGetJsonFromArg(ev, key, SR_OPTIONS, out obj); + + /// <summary> + /// Attempts to read and deserialize a JSON object from the reqeust body (form data or urlencoded) + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="ev"></param> + /// <param name="key">Request argument key (name)</param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during deserialization </param> + /// <param name="obj"></param> + /// <returns>true if the argument was found and successfully converted to json</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidJsonRequestException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool TryGetJsonFromArg<T>(this IHttpEvent ev, string key, JsonSerializerOptions options, out T? obj) + { + //Check for key in argument + if (ev.RequestArgs.TryGetNonEmptyValue(key, out string? value)) + { + try + { + //Deserialize and return the object + obj = value.AsJsonObject<T>(options); + return true; + } + catch(JsonException je) + { + throw new InvalidJsonRequestException(je); + } + } + obj = default; + return false; + } + + /// <summary> + /// Reads the value stored at the key location in the request body arguments, into a <see cref="JsonDocument"/> + /// </summary> + /// <param name="ev"></param> + /// <param name="key">Request argument key (name)</param> + /// <param name="options"><see cref="JsonDocumentOptions"/> to use during parsing</param> + /// <returns>A new <see cref="JsonDocument"/> if the key is found, null otherwise</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="InvalidJsonRequestException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static JsonDocument? GetJsonFromArg(this IHttpEvent ev, string key, in JsonDocumentOptions options = default) + { + try + { + //Check for key in argument + return ev.RequestArgs.TryGetNonEmptyValue(key, out string? value) ? JsonDocument.Parse(value, options) : null; + } + catch (JsonException je) + { + throw new InvalidJsonRequestException(je); + } + } + + /// <summary> + /// If there are file attachements (form data files or content body) and the file is <see cref="ContentType.Json"/> + /// file. It will be deserialzied to the specified object + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="ev"></param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during deserialization </param> + /// <returns>Returns the deserialized object if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidJsonRequestException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T? GetJsonFromFile<T>(this IHttpEvent ev, JsonSerializerOptions? options = null, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return default; + } + + FileUpload file = ev.Files[uploadIndex]; + //Make sure the file is a json file + if (file.ContentType != ContentType.Json) + { + return default; + } + try + { + //Beware this will buffer the entire file object before it attmepts to de-serialize it + return VnEncoding.JSONDeserializeFromBinary<T>(file.FileData, options); + } + catch (JsonException je) + { + throw new InvalidJsonRequestException(je); + } + } + + /// <summary> + /// If there are file attachements (form data files or content body) and the file is <see cref="ContentType.Json"/> + /// file. It will be parsed into a new <see cref="JsonDocument"/> + /// </summary> + /// <param name="ev"></param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <returns>Returns the parsed <see cref="JsonDocument"/>if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidJsonRequestException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static JsonDocument? GetJsonFromFile(this IHttpEvent ev, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return default; + } + FileUpload file = ev.Files[uploadIndex]; + //Make sure the file is a json file + if (file.ContentType != ContentType.Json) + { + return default; + } + try + { + return JsonDocument.Parse(file.FileData); + } + catch(JsonException je) + { + throw new InvalidJsonRequestException(je); + } + } + + /// <summary> + /// If there are file attachements (form data files or content body) and the file is <see cref="ContentType.Json"/> + /// file. It will be deserialzied to the specified object + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="ev"></param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during deserialization </param> + /// <returns>The deserialized object if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + /// <exception cref="InvalidJsonRequestException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ValueTask<T?> GetJsonFromFileAsync<T>(this HttpEntity ev, JsonSerializerOptions? options = null, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return ValueTask.FromResult<T?>(default); + } + FileUpload file = ev.Files[uploadIndex]; + //Make sure the file is a json file + if (file.ContentType != ContentType.Json) + { + return ValueTask.FromResult<T?>(default); + } + //avoid copying the ev struct, so return deserialze task + static async ValueTask<T?> Deserialze(Stream data, JsonSerializerOptions? options, CancellationToken token) + { + try + { + //Beware this will buffer the entire file object before it attmepts to de-serialize it + return await VnEncoding.JSONDeserializeFromBinaryAsync<T?>(data, options, token); + } + catch (JsonException je) + { + throw new InvalidJsonRequestException(je); + } + } + return Deserialze(file.FileData, options, ev.EventCancellation); + } + + static readonly Task<JsonDocument?> DocTaskDefault = Task.FromResult<JsonDocument?>(null); + + /// <summary> + /// If there are file attachements (form data files or content body) and the file is <see cref="ContentType.Json"/> + /// file. It will be parsed into a new <see cref="JsonDocument"/> + /// </summary> + /// <param name="ev"></param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <returns>Returns the parsed <see cref="JsonDocument"/>if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Task<JsonDocument?> GetJsonFromFileAsync(this HttpEntity ev, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return DocTaskDefault; + } + FileUpload file = ev.Files[uploadIndex]; + //Make sure the file is a json file + if (file.ContentType != ContentType.Json) + { + return DocTaskDefault; + } + static async Task<JsonDocument?> Deserialze(Stream data, CancellationToken token) + { + try + { + //Beware this will buffer the entire file object before it attmepts to de-serialize it + return await JsonDocument.ParseAsync(data, cancellationToken: token); + } + catch (JsonException je) + { + throw new InvalidJsonRequestException(je); + } + } + return Deserialze(file.FileData, ev.EventCancellation); + } + + /// <summary> + /// If there are file attachements (form data files or content body) the specified parser will be called to parse the + /// content body asynchronously into a .net object or its default if no attachments are included + /// </summary> + /// <param name="ev"></param> + /// <param name="parser">A function to asynchronously parse the entity body into its object representation</param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <returns>Returns the parsed <typeparamref name="T"/> if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static Task<T?> ParseFileAsAsync<T>(this IHttpEvent ev, Func<Stream, Task<T?>> parser, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return Task.FromResult<T?>(default); + } + //Get the file + FileUpload file = ev.Files[uploadIndex]; + return parser(file.FileData); + } + + /// <summary> + /// If there are file attachements (form data files or content body) the specified parser will be called to parse the + /// content body asynchronously into a .net object or its default if no attachments are included + /// </summary> + /// <param name="ev"></param> + /// <param name="parser">A function to asynchronously parse the entity body into its object representation</param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <returns>Returns the parsed <typeparamref name="T"/> if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static Task<T?> ParseFileAsAsync<T>(this IHttpEvent ev, Func<Stream, string, Task<T?>> parser, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return Task.FromResult<T?>(default); + } + //Get the file + FileUpload file = ev.Files[uploadIndex]; + //Parse the file using the specified parser + return parser(file.FileData, file.ContentTypeString()); + } + + /// <summary> + /// If there are file attachements (form data files or content body) the specified parser will be called to parse the + /// content body asynchronously into a .net object or its default if no attachments are included + /// </summary> + /// <param name="ev"></param> + /// <param name="parser">A function to asynchronously parse the entity body into its object representation</param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <returns>Returns the parsed <typeparamref name="T"/> if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static ValueTask<T?> ParseFileAsAsync<T>(this IHttpEvent ev, Func<Stream, ValueTask<T?>> parser, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return ValueTask.FromResult<T?>(default); + } + //Get the file + FileUpload file = ev.Files[uploadIndex]; + return parser(file.FileData); + } + + /// <summary> + /// If there are file attachements (form data files or content body) the specified parser will be called to parse the + /// content body asynchronously into a .net object or its default if no attachments are included + /// </summary> + /// <param name="ev"></param> + /// <param name="parser">A function to asynchronously parse the entity body into its object representation</param> + /// <param name="uploadIndex">The index within <see cref="HttpEntity.Files"/></param> list of the file to read + /// <returns>Returns the parsed <typeparamref name="T"/> if found, default otherwise</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static ValueTask<T?> ParseFileAsAsync<T>(this IHttpEvent ev, Func<Stream, string, ValueTask<T?>> parser, int uploadIndex = 0) + { + if (ev.Files.Count <= uploadIndex) + { + return ValueTask.FromResult<T?>(default); + } + //Get the file + FileUpload file = ev.Files[uploadIndex]; + //Parse the file using the specified parser + return parser(file.FileData, file.ContentTypeString()); + } + + /// <summary> + /// Gets the bearer token from an authorization header + /// </summary> + /// <param name="ci"></param> + /// <param name="token">The token stored in the user's authorization header</param> + /// <returns>True if the authorization header was set, has a Bearer token value</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HasAuthorization(this IConnectionInfo ci, [NotNullWhen(true)] out string? token) + { + //Get auth header value + string? authorization = ci.Headers[HttpRequestHeader.Authorization]; + //Check if its set + if (!string.IsNullOrWhiteSpace(authorization)) + { + int bearerIndex = authorization.IndexOf(BEARER_STRING, StringComparison.OrdinalIgnoreCase); + //Calc token offset, get token, and trim any whitespace + token = authorization[(bearerIndex + BEARER_LEN)..].Trim(); + return true; + } + token = null; + return false; + } + + /// <summary> + /// Get a <see cref="DirectoryInfo"/> instance that points to the current sites filesystem root. + /// </summary> + /// <returns></returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="PathTooLongException"></exception> + /// <exception cref="ArgumentNullException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static DirectoryInfo GetRootDir(this HttpEntity ev) => new(ev.RequestedRoot.Directory); + + /// <summary> + /// Returns the MIME string representation of the content type of the uploaded file. + /// </summary> + /// <param name="upload"></param> + /// <returns>The MIME string representation of the content type of the uploaded file.</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string ContentTypeString(this in FileUpload upload) => HttpHelpers.GetContentTypeString(upload.ContentType); + + + /// <summary> + /// Attemts to upgrade the connection to a websocket, if the setup fails, it sets up the response to the client accordingly. + /// </summary> + /// <param name="entity"></param> + /// <param name="socketOpenedcallback">A delegate that will be invoked when the websocket has been opened by the framework</param> + /// <param name="subProtocol">The sub-protocol to use on the current websocket</param> + /// <param name="userState">An object to store in the <see cref="WebSocketSession.UserState"/> property when the websocket has been accepted</param> + /// <returns>True if operation succeeds.</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public static bool AcceptWebSocket(this IHttpEvent entity, WebsocketAcceptedCallback socketOpenedcallback, object? userState, string? subProtocol = null) + { + //Make sure this is a websocket request + if (!entity.Server.IsWebSocketRequest) + { + throw new InvalidOperationException("Connection is not a websocket request"); + } + + //Must define an accept callback + _ = socketOpenedcallback ?? throw new ArgumentNullException(nameof(socketOpenedcallback)); + + string? version = entity.Server.Headers["Sec-WebSocket-Version"]; + + //rfc6455:4.2, version must equal 13 + if (!string.IsNullOrWhiteSpace(version) && version.Contains("13", StringComparison.OrdinalIgnoreCase)) + { + //Get socket key + string? key = entity.Server.Headers["Sec-WebSocket-Key"]; + if (!string.IsNullOrWhiteSpace(key) && key.Length < 25) + { + //Set headers for acceptance + entity.Server.Headers[HttpResponseHeader.Upgrade] = "websocket"; + entity.Server.Headers[HttpResponseHeader.Connection] = "Upgrade"; + + //Hash accept string + entity.Server.Headers["Sec-WebSocket-Accept"] = ManagedHash.ComputeBase64Hash($"{key.Trim()}{HttpHelpers.WebsocketRFC4122Guid}", HashAlg.SHA1); + + //Protocol if user specified it + if (!string.IsNullOrWhiteSpace(subProtocol)) + { + entity.Server.Headers["Sec-WebSocket-Protocol"] = subProtocol; + } + + //Setup a new websocket session with a new session id + entity.DangerousChangeProtocol(new WebSocketSession(subProtocol, socketOpenedcallback) + { + IsSecure = entity.Server.IsSecure(), + UserState = userState + }); + + return true; + } + } + //Set the client up for a bad request response, nod a valid websocket request + entity.CloseResponse(HttpStatusCode.BadRequest); + return false; + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/IJsonSerializerBuffer.cs b/Plugins.Essentials/src/Extensions/IJsonSerializerBuffer.cs new file mode 100644 index 0000000..34811f4 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/IJsonSerializerBuffer.cs @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: IJsonSerializerBuffer.cs +* +* IJsonSerializerBuffer.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.IO; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + /// <summary> + /// Interface for a buffer that can be used to serialize objects to JSON + /// </summary> + interface IJsonSerializerBuffer + { + /// <summary> + /// Gets a stream used for writing serialzation data to + /// </summary> + /// <returns>The stream to write JSON data to</returns> + Stream GetSerialzingStream(); + + /// <summary> + /// Called when serialization is complete. + /// The stream may be inspected for the serialized data. + /// </summary> + void SerializationComplete(); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/InternalSerializerExtensions.cs b/Plugins.Essentials/src/Extensions/InternalSerializerExtensions.cs new file mode 100644 index 0000000..3d441a1 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/InternalSerializerExtensions.cs @@ -0,0 +1,100 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: InternalSerializerExtensions.cs +* +* InternalSerializerExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text.Json; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + + internal static class InternalSerializerExtensions + { + + internal static void Serialize<T>(this Utf8JsonWriter writer, IJsonSerializerBuffer buffer, T value, JsonSerializerOptions? options) + { + //Get stream + Stream output = buffer.GetSerialzingStream(); + try + { + //Reset writer + writer.Reset(output); + + //Serialize + JsonSerializer.Serialize(writer, value, options); + + //flush output + writer.Flush(); + } + finally + { + buffer.SerializationComplete(); + } + } + + internal static void Serialize(this Utf8JsonWriter writer, IJsonSerializerBuffer buffer, object value, Type type, JsonSerializerOptions? options) + { + //Get stream + Stream output = buffer.GetSerialzingStream(); + try + { + //Reset writer + writer.Reset(output); + + //Serialize + JsonSerializer.Serialize(writer, value, type, options); + + //flush output + writer.Flush(); + } + finally + { + buffer.SerializationComplete(); + } + } + + internal static void Serialize(this Utf8JsonWriter writer, IJsonSerializerBuffer buffer, JsonDocument document) + { + //Get stream + Stream output = buffer.GetSerialzingStream(); + try + { + //Reset writer + writer.Reset(output); + + //Serialize + document.WriteTo(writer); + + //flush output + writer.Flush(); + } + finally + { + buffer.SerializationComplete(); + } + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/InvalidJsonRequestException.cs b/Plugins.Essentials/src/Extensions/InvalidJsonRequestException.cs new file mode 100644 index 0000000..b2352b2 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/InvalidJsonRequestException.cs @@ -0,0 +1,57 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: InvalidJsonRequestException.cs +* +* InvalidJsonRequestException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Text.Json; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + /// <summary> + /// Wraps a <see cref="JsonException"/> that is thrown when a JSON request message + /// was unsuccessfully parsed. + /// </summary> + public class InvalidJsonRequestException : JsonException + { + /// <summary> + /// Creates a new <see cref="InvalidJsonRequestException"/> wrapper from a base <see cref="JsonException"/> + /// </summary> + /// <param name="baseExp"></param> + public InvalidJsonRequestException(JsonException baseExp) + : base(baseExp.Message, baseExp.Path, baseExp.LineNumber, baseExp.BytePositionInLine, baseExp.InnerException) + { + base.HelpLink = baseExp.HelpLink; + base.Source = baseExp.Source; + } + + public InvalidJsonRequestException() + {} + + public InvalidJsonRequestException(string message) : base(message) + {} + + public InvalidJsonRequestException(string message, System.Exception innerException) : base(message, innerException) + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/JsonResponse.cs b/Plugins.Essentials/src/Extensions/JsonResponse.cs new file mode 100644 index 0000000..22cccd9 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/JsonResponse.cs @@ -0,0 +1,112 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: JsonResponse.cs +* +* JsonResponse.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.IO; + +using VNLib.Net.Http; +using VNLib.Utils.Extensions; +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Memory.Caching; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + internal sealed class JsonResponse : IJsonSerializerBuffer, IMemoryResponseReader + { + private readonly IObjectRental<JsonResponse> _pool; + + private readonly MemoryHandle<byte> _handle; + private readonly IMemoryOwner<byte> _memoryOwner; + //Stream "owns" the handle, so we cannot dispose the stream + private readonly VnMemoryStream _asStream; + + private int _written; + + internal JsonResponse(IObjectRental<JsonResponse> pool) + { + _pool = pool; + + //Alloc buffer + _handle = Memory.Shared.Alloc<byte>(4096, false); + //Consume handle for stream, but make sure not to dispose the stream + _asStream = VnMemoryStream.ConsumeHandle(_handle, 0, false); + //Get memory owner from handle + _memoryOwner = _handle.ToMemoryManager(false); + } + + ~JsonResponse() + { + _handle.Dispose(); + } + + ///<inheritdoc/> + public Stream GetSerialzingStream() + { + //Reset stream position + _asStream.Seek(0, SeekOrigin.Begin); + return _asStream; + } + + ///<inheritdoc/> + public void SerializationComplete() + { + //Reset written position + _written = 0; + //Update remaining pointer + Remaining = Convert.ToInt32(_asStream.Position); + } + + + ///<inheritdoc/> + public int Remaining { get; private set; } + + ///<inheritdoc/> + void IMemoryResponseReader.Advance(int written) + { + //Update position + _written += written; + Remaining -= written; + } + ///<inheritdoc/> + void IMemoryResponseReader.Close() + { + //Reset and return to pool + _written = 0; + Remaining = 0; + //Return self back to pool + _pool.Return(this); + } + + ///<inheritdoc/> + ReadOnlyMemory<byte> IMemoryResponseReader.GetMemory() + { + //Get memory from the memory owner and offet the slice, + return _memoryOwner.Memory.Slice(_written, Remaining); + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/RedirectType.cs b/Plugins.Essentials/src/Extensions/RedirectType.cs new file mode 100644 index 0000000..eff4d38 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/RedirectType.cs @@ -0,0 +1,37 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: RedirectType.cs +* +* RedirectType.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Net; + +namespace VNLib.Plugins.Essentials.Extensions +{ + /// <summary> + /// Shortened list of <see cref="HttpStatusCode"/>s for redirecting connections + /// </summary> + public enum RedirectType + { + None, + Moved = 301, Found = 302, SeeOther = 303, Temporary = 307, Permanent = 308 + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/SimpleMemoryResponse.cs b/Plugins.Essentials/src/Extensions/SimpleMemoryResponse.cs new file mode 100644 index 0000000..a0f2b17 --- /dev/null +++ b/Plugins.Essentials/src/Extensions/SimpleMemoryResponse.cs @@ -0,0 +1,89 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: SimpleMemoryResponse.cs +* +* SimpleMemoryResponse.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using VNLib.Net.Http; +using System.Buffers; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + internal sealed class SimpleMemoryResponse : IMemoryResponseReader + { + private byte[]? _buffer; + private int _written; + + /// <summary> + /// Copies the data in the specified buffer to the internal buffer + /// to initalize the new <see cref="SimpleMemoryResponse"/> + /// </summary> + /// <param name="data">The data to copy</param> + public SimpleMemoryResponse(ReadOnlySpan<byte> data) + { + Remaining = data.Length; + //Alloc buffer + _buffer = ArrayPool<byte>.Shared.Rent(Remaining); + //Copy data to buffer + data.CopyTo(_buffer); + } + + /// <summary> + /// Encodes the character buffer data using the encoder and stores + /// the result in the internal buffer for reading. + /// </summary> + /// <param name="data">The data to encode</param> + /// <param name="enc">The encoder to use</param> + public SimpleMemoryResponse(ReadOnlySpan<char> data, Encoding enc) + { + //Calc byte count + Remaining = enc.GetByteCount(data); + + //Alloc buffer + _buffer = ArrayPool<byte>.Shared.Rent(Remaining); + + //Encode data + Remaining = enc.GetBytes(data, _buffer); + } + + ///<inheritdoc/> + public int Remaining { get; private set; } + ///<inheritdoc/> + void IMemoryResponseReader.Advance(int written) + { + Remaining -= written; + _written += written; + } + ///<inheritdoc/> + void IMemoryResponseReader.Close() + { + //Return buffer to pool + ArrayPool<byte>.Shared.Return(_buffer!); + _buffer = null; + } + ///<inheritdoc/> + ReadOnlyMemory<byte> IMemoryResponseReader.GetMemory() => _buffer!.AsMemory(_written, Remaining); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Extensions/UserExtensions.cs b/Plugins.Essentials/src/Extensions/UserExtensions.cs new file mode 100644 index 0000000..9223b1d --- /dev/null +++ b/Plugins.Essentials/src/Extensions/UserExtensions.cs @@ -0,0 +1,94 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UserExtensions.cs +* +* UserExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; + +using VNLib.Plugins.Essentials.Users; +using VNLib.Plugins.Essentials.Accounts; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Extensions +{ + /// <summary> + /// Provides extension methods to the Users namespace + /// </summary> + public static class UserExtensions + { + + private const string PROFILE_ENTRY = "__.prof"; + + /// <summary> + /// Stores the user's profile to their entry. + /// <br/> + /// NOTE: You must validate/filter data before storing + /// </summary> + /// <param name="ud"></param> + /// <param name="profile">The profile object to store on account</param> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static void SetProfile(this IUser ud, AccountData? profile) + { + //Clear entry if its null + if (profile == null) + { + ud[PROFILE_ENTRY] = null!; + return; + } + //Dont store duplicate values + profile.Created = null; + profile.EmailAddress = null; + ud.SetObject(PROFILE_ENTRY, profile); + } + /// <summary> + /// Stores the serialized string user's profile to their entry. + /// <br/> + /// NOTE: No data validation checks are performed + /// </summary> + /// <param name="ud"></param> + /// <param name="jsonProfile">The JSON serialized "raw" profile data</param> + public static void SetProfile(this IUser ud, string jsonProfile) => ud[PROFILE_ENTRY] = jsonProfile; + /// <summary> + /// Recovers the user's stored profile + /// </summary> + /// <returns>The user's profile stored in the entry or null if no entry is found</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static AccountData? GetProfile(this IUser ud) + { + //Recover profile data, or create new empty profile data + AccountData? ad = ud.GetObject<AccountData>(PROFILE_ENTRY); + if (ad == null) + { + return null; + } + //Set email the same as the account + ad.EmailAddress = ud.EmailAddress; + //Store the rfc time + ad.Created = ud.Created.ToString("R"); + return ad; + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/FileProcessArgs.cs b/Plugins.Essentials/src/FileProcessArgs.cs new file mode 100644 index 0000000..dae695b --- /dev/null +++ b/Plugins.Essentials/src/FileProcessArgs.cs @@ -0,0 +1,169 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: FileProcessArgs.cs +* +* FileProcessArgs.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; + +namespace VNLib.Plugins.Essentials +{ + /// <summary> + /// Server routine to follow after processing selector + /// </summary> + public enum FpRoutine + { + /// <summary> + /// There was an error during processing and the server should immediatly respond with a <see cref="HttpStatusCode.InternalServerError"/> error code + /// </summary> + Error, + /// <summary> + /// The server should continue the file read operation with the current information + /// </summary> + Continue, + /// <summary> + /// The server should redirect the conneciton to an alternate location + /// </summary> + Redirect, + /// <summary> + /// The server should immediatly respond with a <see cref="HttpStatusCode.Forbidden"/> error code + /// </summary> + Deny, + /// <summary> + /// The server should fulfill the reqeest by sending the contents of an alternate file location (if it exists) with the existing connection + /// </summary> + ServeOther, + /// <summary> + /// The server should immediatly respond with a <see cref="HttpStatusCode.NotFound"/> error code + /// </summary> + NotFound, + /// <summary> + /// Serves another file location that must be a trusted fully qualified location + /// </summary> + ServeOtherFQ, + /// <summary> + /// The connection does not require a file to be processed + /// </summary> + VirtualSkip, + } + + /// <summary> + /// Specifies operations the file processor will follow during request handling + /// </summary> + public readonly struct FileProcessArgs : IEquatable<FileProcessArgs> + { + /// <summary> + /// Signals the file processor should complete with a <see cref="FpRoutine.Deny"/> routine + /// </summary> + public static readonly FileProcessArgs Deny = new (FpRoutine.Deny); + /// <summary> + /// Signals the file processor should continue with intended/normal processing of the request + /// </summary> + public static readonly FileProcessArgs Continue = new (FpRoutine.Continue); + /// <summary> + /// Signals the file processor should complete with a <see cref="FpRoutine.Error"/> routine + /// </summary> + public static readonly FileProcessArgs Error = new (FpRoutine.Error); + /// <summary> + /// Signals the file processor should complete with a <see cref="FpRoutine.NotFound"/> routine + /// </summary> + public static readonly FileProcessArgs NotFound = new (FpRoutine.NotFound); + /// <summary> + /// Signals the file processor should not process the connection + /// </summary> + public static readonly FileProcessArgs VirtualSkip = new (FpRoutine.VirtualSkip); + /// <summary> + /// The routine the file processor should execute + /// </summary> + public readonly FpRoutine Routine { get; init; } + /// <summary> + /// An optional alternate path for the given routine + /// </summary> + public readonly string Alternate { get; init; } + + /// <summary> + /// Initializes a new <see cref="FileProcessArgs"/> with the specified routine + /// and empty <see cref="Alternate"/> path + /// </summary> + /// <param name="routine">The file processing routine to execute</param> + public FileProcessArgs(FpRoutine routine) + { + this.Routine = routine; + this.Alternate = string.Empty; + } + /// <summary> + /// Initializes a new <see cref="FileProcessArgs"/> with the specified routine + /// and alternate path + /// </summary> + /// <param name="routine"></param> + /// <param name="alternatePath"></param> + public FileProcessArgs(FpRoutine routine, string alternatePath) + { + this.Routine = routine; + this.Alternate = alternatePath; + } + /// <summary> + /// + /// </summary> + /// <param name="arg1"></param> + /// <param name="arg2"></param> + /// <returns></returns> + public static bool operator == (FileProcessArgs arg1, FileProcessArgs arg2) + { + return arg1.Equals(arg2); + } + /// <summary> + /// + /// </summary> + /// <param name="arg1"></param> + /// <param name="arg2"></param> + /// <returns></returns> + public static bool operator != (FileProcessArgs arg1, FileProcessArgs arg2) + { + return !arg1.Equals(arg2); + } + ///<inheritdoc/> + public bool Equals(FileProcessArgs other) + { + //make sure the routine types match + if (Routine != other.Routine) + { + return false; + } + //Next make sure the hashcodes of the alternate paths match + return (Alternate?.GetHashCode(StringComparison.OrdinalIgnoreCase)) == (other.Alternate?.GetHashCode(StringComparison.OrdinalIgnoreCase)); + } + ///<inheritdoc/> + public override bool Equals(object obj) + { + return obj is FileProcessArgs args && Equals(args); + } + /// <summary> + /// <inheritdoc/> + /// </summary> + /// <returns></returns> + public override int GetHashCode() + { + return base.GetHashCode(); + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/HttpEntity.cs b/Plugins.Essentials/src/HttpEntity.cs new file mode 100644 index 0000000..ffad607 --- /dev/null +++ b/Plugins.Essentials/src/HttpEntity.cs @@ -0,0 +1,178 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: HttpEntity.cs +* +* HttpEntity.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Threading; +using System.Collections.Generic; +using System.Runtime.CompilerServices; + +using VNLib.Net.Http; +using VNLib.Plugins.Essentials.Sessions; +using VNLib.Plugins.Essentials.Extensions; + +#nullable enable + +/* + * HttpEntity was converted to an object as during profiling + * it was almost always heap allcated due to async opertaions + * or other object tracking issues. So to reduce the number of + * allocations (at the cost of larger objects) basic profiling + * showed less GC load and less collections when SessionInfo + * remained a value type + */ +#pragma warning disable CA1051 // Do not declare visible instance fields + +namespace VNLib.Plugins.Essentials +{ + /// <summary> + /// A container for an <see cref="HttpEvent"/> with its attached session. + /// This class cannot be inherited. + /// </summary> + public sealed class HttpEntity : IHttpEvent + { + /// <summary> + /// The connection event entity + /// </summary> + private readonly IHttpEvent Entity; + public HttpEntity(IHttpEvent entity, EventProcessor root, in SessionHandle session, in CancellationToken cancellation) + { + Entity = entity; + RequestedRoot = root; + EventCancellation = cancellation; + + //See if the connection is coming from an downstream server + IsBehindDownStreamServer = root.Options.DownStreamServers.Contains(entity.Server.RemoteEndpoint.Address); + /* + * If the connection was behind a trusted downstream server, + * we can trust the x-forwarded-for header, + * otherwise use the remote ep ip address + */ + TrustedRemoteIp = entity.Server.GetTrustedIp(IsBehindDownStreamServer); + //Initialize the session + Session = session.IsSet ? new(session.SessionData, entity.Server, TrustedRemoteIp) : new(); + //Local connection + IsLocalConnection = entity.Server.LocalEndpoint.Address.IsLocalSubnet(TrustedRemoteIp); + //Cache value + IsSecure = entity.Server.IsSecure(IsBehindDownStreamServer); + } + + /// <summary> + /// A token that has a scheduled timeout to signal the cancellation of the entity event + /// </summary> + public readonly CancellationToken EventCancellation; + /// <summary> + /// The session assocaited with the event + /// </summary> + public readonly SessionInfo Session; + /// <summary> + /// A value that indicates if the connecion came from a trusted downstream server + /// </summary> + public readonly bool IsBehindDownStreamServer; + /// <summary> + /// Determines if the connection came from the local network to the current server + /// </summary> + public readonly bool IsLocalConnection; + /// <summary> + /// Gets a value that determines if the connection is using tls, locally + /// or behind a trusted downstream server that is using tls. + /// </summary> + public readonly bool IsSecure; + + /// <summary> + /// The connection info object assocated with the entity + /// </summary> + public IConnectionInfo Server => Entity.Server; + /// <summary> + /// User's ip. If the connection is behind a local proxy, returns the users actual IP. Otherwise returns the connection ip. + /// </summary> + public readonly IPAddress TrustedRemoteIp; + /// <summary> + /// The requested web root. Provides additional site information + /// </summary> + public readonly EventProcessor RequestedRoot; + /// <summary> + /// If the request has query arguments they are stored in key value format + /// </summary> + public IReadOnlyDictionary<string, string> QueryArgs => Entity.QueryArgs; + /// <summary> + /// If the request body has form data or url encoded arguments they are stored in key value format + /// </summary> + public IReadOnlyDictionary<string, string> RequestArgs => Entity.RequestArgs; + /// <summary> + /// Contains all files upladed with current request + /// </summary> + public IReadOnlyList<FileUpload> Files => Entity.Files; + ///<inheritdoc/> + HttpServer IHttpEvent.OriginServer => Entity.OriginServer; + + /// <summary> + /// Complete the session and respond to user + /// </summary> + /// <param name="code">Status code of operation</param> + /// <exception cref="InvalidOperationException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void CloseResponse(HttpStatusCode code) => Entity.CloseResponse(code); + + ///<inheritdoc/> + ///<exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void CloseResponse(HttpStatusCode code, ContentType type, Stream stream) + { + Entity.CloseResponse(code, type, stream); + //Verify content type matches + if (!Server.Accepts(type)) + { + throw new ContentTypeUnacceptableException("The client does not accept the content type of the response"); + } + } + + ///<inheritdoc/> + ///<exception cref="ContentTypeUnacceptableException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void CloseResponse(HttpStatusCode code, ContentType type, IMemoryResponseReader entity) + { + //Verify content type matches + if (!Server.Accepts(type)) + { + throw new ContentTypeUnacceptableException("The client does not accept the content type of the response"); + } + + Entity.CloseResponse(code, type, entity); + } + + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void DisableCompression() => Entity.DisableCompression(); + + /* + * Do not directly expose dangerous methods, but allow them to be called + */ + + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + void IHttpEvent.DangerousChangeProtocol(IAlternateProtocol protocolHandler) => Entity.DangerousChangeProtocol(protocolHandler); + } +} diff --git a/Plugins.Essentials/src/IEpProcessingOptions.cs b/Plugins.Essentials/src/IEpProcessingOptions.cs new file mode 100644 index 0000000..de79327 --- /dev/null +++ b/Plugins.Essentials/src/IEpProcessingOptions.cs @@ -0,0 +1,66 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: IEpProcessingOptions.cs +* +* IEpProcessingOptions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Collections.Generic; + +#nullable enable + +namespace VNLib.Plugins.Essentials +{ + /// <summary> + /// Provides an interface for <see cref="EventProcessor"/> + /// security options + /// </summary> + public interface IEpProcessingOptions + { + /// <summary> + /// The name of a default file to search for within a directory if no file is specified (index.html). + /// This array should be ordered. + /// </summary> + IReadOnlyCollection<string> DefaultFiles { get; } + /// <summary> + /// File extensions that are denied from being read from the filesystem + /// </summary> + IReadOnlySet<string> ExcludedExtensions { get; } + /// <summary> + /// File attributes that must be matched for the file to be accessed + /// </summary> + FileAttributes AllowedAttributes { get; } + /// <summary> + /// Files that match any attribute flag set will be denied + /// </summary> + FileAttributes DissallowedAttributes { get; } + /// <summary> + /// A table of known downstream servers/ports that can be trusted to proxy connections + /// </summary> + IReadOnlySet<IPAddress> DownStreamServers { get; } + /// <summary> + /// A <see cref="TimeSpan"/> for how long a connection may remain open before all operations are cancelled + /// </summary> + TimeSpan ExecutionTimeout { get; } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Oauth/IOAuth2Provider.cs b/Plugins.Essentials/src/Oauth/IOAuth2Provider.cs new file mode 100644 index 0000000..30944b8 --- /dev/null +++ b/Plugins.Essentials/src/Oauth/IOAuth2Provider.cs @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: IOAuth2Provider.cs +* +* IOAuth2Provider.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading.Tasks; + +using VNLib.Plugins.Essentials.Sessions; + +namespace VNLib.Plugins.Essentials.Oauth +{ + /// <summary> + /// An interface that Oauth2 serice providers must implement + /// to provide sessions to an <see cref="EventProcessor"/> + /// processor endpoint processor + /// </summary> + public interface IOAuth2Provider : ISessionProvider + { + /// <summary> + /// Gets a value indicating how long a session may be valid for + /// </summary> + public TimeSpan MaxTokenLifetime { get; } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Oauth/O2EndpointBase.cs b/Plugins.Essentials/src/Oauth/O2EndpointBase.cs new file mode 100644 index 0000000..a1a4d35 --- /dev/null +++ b/Plugins.Essentials/src/Oauth/O2EndpointBase.cs @@ -0,0 +1,162 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: O2EndpointBase.cs +* +* O2EndpointBase.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Net; +using System.Net.Sockets; +using System.Threading.Tasks; + +using VNLib.Utils; +using VNLib.Utils.Logging; +using VNLib.Net.Http; +using VNLib.Plugins.Essentials.Extensions; +using VNLib.Plugins.Essentials.Endpoints; + +namespace VNLib.Plugins.Essentials.Oauth +{ + /// <summary> + /// An base class for HttpEntity processors (endpoints) for processing + /// Oauth2 client requests. Similar to <seealso cref="ProtectedWebEndpoint"/> + /// but for Oauth2 sessions + /// </summary> + public abstract class O2EndpointBase : ResourceEndpointBase + { + //Disable browser only protection + ///<inheritdoc/> + protected override ProtectionSettings EndpointProtectionSettings { get; } = new() { DisableBrowsersOnly = true }; + + ///<inheritdoc/> + public override async ValueTask<VfReturnType> Process(HttpEntity entity) + { + try + { + VfReturnType rt; + ERRNO preProc = PreProccess(entity); + //Entity was responded to by the pre-processor + if (preProc < 0) + { + return VfReturnType.VirtualSkip; + } + if (preProc == ERRNO.E_FAIL) + { + rt = VfReturnType.Forbidden; + goto Exit; + } + //If websockets are quested allow them to be processed in a logged-in/secure context + if (entity.Server.IsWebSocketRequest) + { + return await WebsocketRequestedAsync(entity); + } + //Capture return type + rt = entity.Server.Method switch + { + //Get request to get account + HttpMethod.GET => await GetAsync(entity), + HttpMethod.POST => await PostAsync(entity), + HttpMethod.DELETE => await DeleteAsync(entity), + HttpMethod.PUT => await PutAsync(entity), + HttpMethod.PATCH => await PatchAsync(entity), + HttpMethod.OPTIONS => await OptionsAsync(entity), + _ => await AlternateMethodAsync(entity, entity.Server.Method), + }; + Exit: + //Write a standard Ouath2 error messag + switch (rt) + { + case VfReturnType.VirtualSkip: + return VfReturnType.VirtualSkip; + case VfReturnType.ProcessAsFile: + return VfReturnType.ProcessAsFile; + case VfReturnType.NotFound: + entity.CloseResponseError(HttpStatusCode.NotFound, ErrorType.InvalidRequest, "The requested resource could not be found"); + return VfReturnType.VirtualSkip; + case VfReturnType.BadRequest: + entity.CloseResponseError(HttpStatusCode.BadRequest, ErrorType.InvalidRequest, "Your request was not properlty formatted and could not be proccessed"); + return VfReturnType.VirtualSkip; + case VfReturnType.Error: + entity.CloseResponseError(HttpStatusCode.InternalServerError, ErrorType.ServerError, "There was a server error processing your request"); + return VfReturnType.VirtualSkip; + + case VfReturnType.Forbidden: + default: + entity.CloseResponseError(HttpStatusCode.Forbidden, ErrorType.InvalidClient, "You do not have access to this resource"); + return VfReturnType.VirtualSkip; + } + } + catch (TerminateConnectionException) + { + //A TC exception is intentional and should always propagate to the runtime + throw; + } + //Re-throw exceptions that are cause by reading the transport layer + catch (IOException ioe) when (ioe.InnerException is SocketException) + { + throw; + } + catch (ContentTypeUnacceptableException) + { + //Respond with an 406 error message + entity.CloseResponseError(HttpStatusCode.NotAcceptable, ErrorType.InvalidRequest, "The response type is not acceptable for this endpoint"); + return VfReturnType.VirtualSkip; + } + catch (InvalidJsonRequestException) + { + //Respond with an error message + entity.CloseResponseError(HttpStatusCode.BadRequest, ErrorType.InvalidRequest, "The request body was not a proper JSON schema"); + return VfReturnType.VirtualSkip; + } + catch (Exception ex) + { + //Log an uncaught excetpion and return an error code (log may not be initialized) + Log?.Error(ex); + //Respond with an error message + entity.CloseResponseError(HttpStatusCode.InternalServerError, ErrorType.ServerError, "There was a server error processing your request"); + return VfReturnType.VirtualSkip; + } + } + + /// <summary> + /// Runs base pre-processing and ensures "sessions" OAuth2 token + /// session is loaded + /// </summary> + /// <param name="entity">The request entity to process</param> + /// <inheritdoc/> + protected override ERRNO PreProccess(HttpEntity entity) + { + //Make sure session is loaded (token is valid) + if (!entity.Session.IsSet) + { + entity.CloseResponseError(HttpStatusCode.Forbidden, ErrorType.InvalidToken, "Your token is not valid"); + return -1; + } + //Must be an oauth session + if (entity.Session.SessionType != Sessions.SessionType.OAuth2) + { + return false; + } + return base.PreProccess(entity); + } + } +} diff --git a/Plugins.Essentials/src/Oauth/OauthHttpExtensions.cs b/Plugins.Essentials/src/Oauth/OauthHttpExtensions.cs new file mode 100644 index 0000000..892a24c --- /dev/null +++ b/Plugins.Essentials/src/Oauth/OauthHttpExtensions.cs @@ -0,0 +1,239 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: OauthHttpExtensions.cs +* +* OauthHttpExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System.Net; +using System.Text; + +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Memory.Caching; +using VNLib.Plugins.Essentials.Extensions; + +namespace VNLib.Plugins.Essentials.Oauth +{ + /// <summary> + /// An OAuth2 specification error code + /// </summary> + public enum ErrorType + { + /// <summary> + /// The request is considered invalid and cannot be continued + /// </summary> + InvalidRequest, + /// <summary> + /// + /// </summary> + InvalidClient, + /// <summary> + /// The supplied token is no longer considered valid + /// </summary> + InvalidToken, + /// <summary> + /// The token does not have the authorization required, is missing authorization, or is no longer considered acceptable + /// </summary> + UnauthorizedClient, + /// <summary> + /// The client accept content type is unacceptable for the requested endpoint and cannot be processed + /// </summary> + UnsupportedResponseType, + /// <summary> + /// The scope of the token does not allow for this operation + /// </summary> + InvalidScope, + /// <summary> + /// There was a server related error and the request could not be fulfilled + /// </summary> + ServerError, + /// <summary> + /// The request could not be processed at this time + /// </summary> + TemporarilyUnabavailable + } + + public static class OauthHttpExtensions + { + private static ThreadLocalObjectStorage<StringBuilder> SbRental { get; } = ObjectRental.CreateThreadLocal(Constructor, null, ReturnFunc); + + private static StringBuilder Constructor() => new(64); + private static void ReturnFunc(StringBuilder sb) => sb.Clear(); + + /// <summary> + /// Closes the current response with a json error message with the message details + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The http status code</param> + /// <param name="error">The short error</param> + /// <param name="description">The error description message</param> + public static void CloseResponseError(this HttpEntity ev, HttpStatusCode code, ErrorType error, string description) + { + //See if the response accepts json + if (ev.Server.Accepts(ContentType.Json)) + { + //Use a stringbuilder to create json result for the error description + StringBuilder sb = SbRental.Rent(); + sb.Append("{\"error\":\""); + switch (error) + { + case ErrorType.InvalidRequest: + sb.Append("invalid_request"); + break; + case ErrorType.InvalidClient: + sb.Append("invalid_client"); + break; + case ErrorType.UnauthorizedClient: + sb.Append("unauthorized_client"); + break; + case ErrorType.InvalidToken: + sb.Append("invalid_token"); + break; + case ErrorType.UnsupportedResponseType: + sb.Append("unsupported_response_type"); + break; + case ErrorType.InvalidScope: + sb.Append("invalid_scope"); + break; + case ErrorType.ServerError: + sb.Append("server_error"); + break; + case ErrorType.TemporarilyUnabavailable: + sb.Append("temporarily_unavailable"); + break; + default: + sb.Append("error"); + break; + } + sb.Append("\",\"error_description\":\""); + sb.Append(description); + sb.Append("\"}"); + //Close the response with the json data + ev.CloseResponse(code, ContentType.Json, sb.ToString()); + //Return the builder + SbRental.Return(sb); + } + //Otherwise set the error code in the wwwauth header + else + { + //Set the error result in the header + ev.Server.Headers[HttpResponseHeader.WwwAuthenticate] = error switch + { + ErrorType.InvalidRequest => $"Bearer error=\"invalid_request\"", + ErrorType.UnauthorizedClient => $"Bearer error=\"unauthorized_client\"", + ErrorType.UnsupportedResponseType => $"Bearer error=\"unsupported_response_type\"", + ErrorType.InvalidScope => $"Bearer error=\"invalid_scope\"", + ErrorType.ServerError => $"Bearer error=\"server_error\"", + ErrorType.TemporarilyUnabavailable => $"Bearer error=\"temporarily_unavailable\"", + ErrorType.InvalidClient => $"Bearer error=\"invalid_client\"", + ErrorType.InvalidToken => $"Bearer error=\"invalid_token\"", + _ => $"Bearer error=\"error\"", + }; + //Close the response with the status code + ev.CloseResponse(code); + } + } + /// <summary> + /// Closes the current response with a json error message with the message details + /// </summary> + /// <param name="ev"></param> + /// <param name="code">The http status code</param> + /// <param name="error">The short error</param> + /// <param name="description">The error description message</param> + public static void CloseResponseError(this IHttpEvent ev, HttpStatusCode code, ErrorType error, string description) + { + //See if the response accepts json + if (ev.Server.Accepts(ContentType.Json)) + { + //Use a stringbuilder to create json result for the error description + StringBuilder sb = SbRental.Rent(); + sb.Append("{\"error\":\""); + switch (error) + { + case ErrorType.InvalidRequest: + sb.Append("invalid_request"); + break; + case ErrorType.InvalidClient: + sb.Append("invalid_client"); + break; + case ErrorType.UnauthorizedClient: + sb.Append("unauthorized_client"); + break; + case ErrorType.InvalidToken: + sb.Append("invalid_token"); + break; + case ErrorType.UnsupportedResponseType: + sb.Append("unsupported_response_type"); + break; + case ErrorType.InvalidScope: + sb.Append("invalid_scope"); + break; + case ErrorType.ServerError: + sb.Append("server_error"); + break; + case ErrorType.TemporarilyUnabavailable: + sb.Append("temporarily_unavailable"); + break; + default: + sb.Append("error"); + break; + } + sb.Append("\",\"error_description\":\""); + sb.Append(description); + sb.Append("\"}"); + + VnMemoryStream vms = VnEncoding.GetMemoryStream(sb.ToString(), ev.Server.Encoding); + try + { + //Close the response with the json data + ev.CloseResponse(code, ContentType.Json, vms); + } + catch + { + vms.Dispose(); + throw; + } + //Return the builder + SbRental.Return(sb); + } + //Otherwise set the error code in the wwwauth header + else + { + //Set the error result in the header + ev.Server.Headers[HttpResponseHeader.WwwAuthenticate] = error switch + { + ErrorType.InvalidRequest => $"Bearer error=\"invalid_request\"", + ErrorType.UnauthorizedClient => $"Bearer error=\"unauthorized_client\"", + ErrorType.UnsupportedResponseType => $"Bearer error=\"unsupported_response_type\"", + ErrorType.InvalidScope => $"Bearer error=\"invalid_scope\"", + ErrorType.ServerError => $"Bearer error=\"server_error\"", + ErrorType.TemporarilyUnabavailable => $"Bearer error=\"temporarily_unavailable\"", + ErrorType.InvalidClient => $"Bearer error=\"invalid_client\"", + ErrorType.InvalidToken => $"Bearer error=\"invalid_token\"", + _ => $"Bearer error=\"error\"", + }; + //Close the response with the status code + ev.CloseResponse(code); + } + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Oauth/OauthSessionCacheExhaustedException.cs b/Plugins.Essentials/src/Oauth/OauthSessionCacheExhaustedException.cs new file mode 100644 index 0000000..da91444 --- /dev/null +++ b/Plugins.Essentials/src/Oauth/OauthSessionCacheExhaustedException.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: OauthSessionCacheExhaustedException.cs +* +* OauthSessionCacheExhaustedException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + + +namespace VNLib.Plugins.Essentials.Oauth +{ + /// <summary> + /// Raised when the session cache space has been exhausted and cannot + /// load the new session into cache. + /// </summary> + public class OauthSessionCacheExhaustedException : Exception + { + public OauthSessionCacheExhaustedException(string message) : base(message) + {} + public OauthSessionCacheExhaustedException(string message, Exception innerException) : base(message, innerException) + {} + public OauthSessionCacheExhaustedException() + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Oauth/OauthSessionExtensions.cs b/Plugins.Essentials/src/Oauth/OauthSessionExtensions.cs new file mode 100644 index 0000000..6f9d275 --- /dev/null +++ b/Plugins.Essentials/src/Oauth/OauthSessionExtensions.cs @@ -0,0 +1,88 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: OauthSessionExtensions.cs +* +* OauthSessionExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Plugins.Essentials.Sessions; + +namespace VNLib.Plugins.Essentials.Oauth +{ + /// <summary> + /// Represents an active oauth session + /// </summary> + public static class OauthSessionExtensions + { + public const string APP_ID_ENTRY = "oau.apid"; + public const string REFRESH_TOKEN_ENTRY = "oau.rt"; + public const string SCOPES_ENTRY = "oau.scp"; + public const string TOKEN_TYPE_ENTRY = "oau.typ"; + + + /// <summary> + /// The ID of the application that granted the this token access + /// </summary> + public static string AppID(this in SessionInfo session) => session[APP_ID_ENTRY]; + + /// <summary> + /// The refresh token for this current token + /// </summary> + public static string RefreshToken(this in SessionInfo session) => session[REFRESH_TOKEN_ENTRY]; + + /// <summary> + /// The token's privilage scope + /// </summary> + public static string Scopes(this in SessionInfo session) => session[SCOPES_ENTRY]; + /// <summary> + /// The Oauth2 token type + /// </summary>, + public static string Type(this in SessionInfo session) => session[TOKEN_TYPE_ENTRY]; + + /// <summary> + /// Determines if the current session has the required scope type and the + /// specified permission + /// </summary> + /// <param name="session"></param> + /// <param name="type">The scope type</param> + /// <param name="permission">The scope permission</param> + /// <returns>True if the current session has the required scope, false otherwise</returns> + public static bool HasScope(this in SessionInfo session, string type, string permission) + { + //Join the permissions components + string perms = string.Concat(type, ":", permission); + return session.HasScope(perms); + } + /// <summary> + /// Determines if the current session has the required scope type and the + /// specified permission + /// </summary> + /// <param name="session"></param> + /// <param name="scope">The scope to compare</param> + /// <returns>True if the current session has the required scope, false otherwise</returns> + public static bool HasScope(this in SessionInfo session, ReadOnlySpan<char> scope) + { + //Split the scope components and check them against the joined permission + return session.Scopes().AsSpan().Contains(scope, StringComparison.OrdinalIgnoreCase); + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Sessions/ISession.cs b/Plugins.Essentials/src/Sessions/ISession.cs new file mode 100644 index 0000000..e15c6e2 --- /dev/null +++ b/Plugins.Essentials/src/Sessions/ISession.cs @@ -0,0 +1,94 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ISession.cs +* +* ISession.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; + +using VNLib.Utils; + +namespace VNLib.Plugins.Essentials.Sessions +{ + /// <summary> + /// Flags to specify <see cref="ISession"/> session types + /// </summary> + public enum SessionType + { + /// <summary> + /// The session is a "basic" or web based session + /// </summary> + Web, + /// <summary> + /// The session is an OAuth2 session type + /// </summary> + OAuth2 + } + + /// <summary> + /// Represents a connection oriented session data + /// </summary> + public interface ISession : IIndexable<string, string> + { + /// <summary> + /// A value specifying the type of the loaded session + /// </summary> + SessionType SessionType { get; } + /// <summary> + /// UTC time in when the session was created + /// </summary> + DateTimeOffset Created { get; } + /// <summary> + /// Privilages associated with user specified during login + /// </summary> + ulong Privilages { get; set; } + /// <summary> + /// Key that identifies the current session. (Identical to cookie::sessionid) + /// </summary> + string SessionID { get; } + /// <summary> + /// User ID associated with session + /// </summary> + string UserID { get; set; } + /// <summary> + /// Marks the session as invalid + /// </summary> + void Invalidate(bool all = false); + /// <summary> + /// Gets or sets the session's authorization token + /// </summary> + string Token { get; set; } + /// <summary> + /// The IP address belonging to the client + /// </summary> + IPAddress UserIP { get; } + /// <summary> + /// Sets the session ID to be regenerated if applicable + /// </summary> + void RegenID(); + + /// <summary> + /// A value that indicates this session was newly created + /// </summary> + bool IsNew { get; } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Sessions/ISessionExtensions.cs b/Plugins.Essentials/src/Sessions/ISessionExtensions.cs new file mode 100644 index 0000000..44063f9 --- /dev/null +++ b/Plugins.Essentials/src/Sessions/ISessionExtensions.cs @@ -0,0 +1,95 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ISessionExtensions.cs +* +* ISessionExtensions.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.CompilerServices; +using System.Security.Authentication; + +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.Extensions; + +namespace VNLib.Plugins.Essentials.Sessions +{ + public static class ISessionExtensions + { + public const string USER_AGENT_ENTRY = "__.ua"; + public const string ORIGIN_ENTRY = "__.org"; + public const string REFER_ENTRY = "__.rfr"; + public const string SECURE_ENTRY = "__.sec"; + public const string CROSS_ORIGIN = "__.cor"; + public const string LOCAL_TIME_ENTRY = "__.lot"; + public const string LOGIN_TOKEN_ENTRY = "__.lte"; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetUserAgent(this ISession session) => session[USER_AGENT_ENTRY]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetUserAgent(this ISession session, string userAgent) => session[USER_AGENT_ENTRY] = userAgent; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetOrigin(this ISession session) => session[ORIGIN_ENTRY]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Uri GetOriginUri(this ISession session) => Uri.TryCreate(session[ORIGIN_ENTRY], UriKind.Absolute, out Uri origin) ? origin : null; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetOrigin(this ISession session, string origin) => session[ORIGIN_ENTRY] = origin; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetRefer(this ISession session) => session[REFER_ENTRY]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetRefer(this ISession session, string refer) => session[REFER_ENTRY] = refer; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static SslProtocols GetSecurityProtocol(this ISession session) + { + return (SslProtocols)session.GetValueType<string, int>(SECURE_ENTRY); + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetSecurityProtocol(this ISession session, SslProtocols protocol) => session[SECURE_ENTRY] = VnEncoding.ToBase32String((int)protocol); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool IsCrossOrigin(this ISession session) => session[CROSS_ORIGIN] == bool.TrueString; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void IsCrossOrigin(this ISession session, bool crossOrign) => session[CROSS_ORIGIN] = crossOrign.ToString(); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string GetLoginToken(this ISession session) => session[LOGIN_TOKEN_ENTRY]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void SetLoginToken(this ISession session, string loginToken) => session[LOGIN_TOKEN_ENTRY] = loginToken; + + /// <summary> + /// Initializes a "new" session with initial varaibles from the current connection + /// for lookup/comparison later + /// </summary> + /// <param name="session"></param> + /// <param name="ci">The <see cref="ConnectionInfo"/> object containing connection details</param> + public static void InitNewSession(this ISession session, IConnectionInfo ci) + { + session.IsCrossOrigin(ci.CrossOrigin); + session.SetOrigin(ci.Origin?.ToString()); + session.SetRefer(ci.Referer?.ToString()); + session.SetSecurityProtocol(ci.SecurityProtocol); + session.SetUserAgent(ci.UserAgent); + } + + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Sessions/ISessionProvider.cs b/Plugins.Essentials/src/Sessions/ISessionProvider.cs new file mode 100644 index 0000000..fe7e7ce --- /dev/null +++ b/Plugins.Essentials/src/Sessions/ISessionProvider.cs @@ -0,0 +1,49 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: ISessionProvider.cs +* +* ISessionProvider.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Net.Http; + +namespace VNLib.Plugins.Essentials.Sessions +{ + /// <summary> + /// Provides stateful session objects assocated with HTTP connections + /// </summary> + public interface ISessionProvider + { + /// <summary> + /// Gets a session handle for the current connection + /// </summary> + /// <param name="entity">The connection to get associated session on</param> + /// <param name="cancellationToken"></param> + /// <returns>A task the resolves an <see cref="SessionHandle"/> instance</returns> + /// <exception cref="TimeoutException"></exception> + /// <exception cref="SessionException"></exception> + /// <exception cref="OperationCanceledException"></exception> + public ValueTask<SessionHandle> GetSessionAsync(IHttpEvent entity, CancellationToken cancellationToken); + } +} diff --git a/Plugins.Essentials/src/Sessions/SessionBase.cs b/Plugins.Essentials/src/Sessions/SessionBase.cs new file mode 100644 index 0000000..d386b8b --- /dev/null +++ b/Plugins.Essentials/src/Sessions/SessionBase.cs @@ -0,0 +1,168 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: SessionBase.cs +* +* SessionBase.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Runtime.CompilerServices; + +using VNLib.Net.Http; +using VNLib.Utils; +using VNLib.Utils.Async; + +namespace VNLib.Plugins.Essentials.Sessions +{ + /// <summary> + /// Provides a base class for the <see cref="ISession"/> interface for exclusive use within a multithreaded + /// context + /// </summary> + public abstract class SessionBase : AsyncExclusiveResource<IHttpEvent>, ISession + { + protected const ulong MODIFIED_MSK = 0b0000000000000001UL; + protected const ulong IS_NEW_MSK = 0b0000000000000010UL; + protected const ulong REGEN_ID_MSK = 0b0000000000000100UL; + protected const ulong INVALID_MSK = 0b0000000000001000UL; + protected const ulong ALL_INVALID_MSK = 0b0000000000100000UL; + + protected const string USER_ID_ENTRY = "__.i.uid"; + protected const string TOKEN_ENTRY = "__.i.tk"; + protected const string PRIV_ENTRY = "__.i.pl"; + protected const string IP_ADDRESS_ENTRY = "__.i.uip"; + protected const string SESSION_TYPE_ENTRY = "__.i.tp"; + + /// <summary> + /// A <see cref="BitField"/> of status flags for the state of the current session. + /// May be used internally + /// </summary> + protected BitField Flags { get; } = new(0); + + /// <summary> + /// Gets or sets the Modified flag + /// </summary> + protected bool IsModified + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => Flags.IsSet(MODIFIED_MSK); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => Flags.Set(MODIFIED_MSK, value); + } + + ///<inheritdoc/> + public virtual string SessionID { get; protected set; } + ///<inheritdoc/> + public virtual DateTimeOffset Created { get; protected set; } + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public string this[string index] + { + get + { + Check(); + return IndexerGet(index); + } + set + { + Check(); + IndexerSet(index, value); + } + } + ///<inheritdoc/> + public virtual IPAddress UserIP + { + get + { + //try to parse the IP address, otherwise return null + _ = IPAddress.TryParse(this[IP_ADDRESS_ENTRY], out IPAddress ip); + return ip; + } + protected set + { + //Store the IP address as its string representation + this[IP_ADDRESS_ENTRY] = value?.ToString(); + } + } + ///<inheritdoc/> + public virtual SessionType SessionType + { + get => Enum.Parse<SessionType>(this[SESSION_TYPE_ENTRY]); + protected set => this[SESSION_TYPE_ENTRY] = ((byte)value).ToString(); + } + + ///<inheritdoc/> + public virtual ulong Privilages + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => Convert.ToUInt64(this[PRIV_ENTRY], 16); + //Store in hexadecimal to conserve space + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => this[PRIV_ENTRY] = value.ToString("X"); + } + ///<inheritdoc/> + public bool IsNew + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => Flags.IsSet(IS_NEW_MSK); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => Flags.Set(IS_NEW_MSK, value); + } + ///<inheritdoc/> + public virtual string UserID + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => this[USER_ID_ENTRY]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => this[USER_ID_ENTRY] = value; + } + ///<inheritdoc/> + public virtual string Token + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => this[TOKEN_ENTRY]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => this[TOKEN_ENTRY] = value; + } + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public virtual void Invalidate(bool all = false) + { + Flags.Set(INVALID_MSK); + Flags.Set(ALL_INVALID_MSK, all); + } + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public virtual void RegenID() => Flags.Set(REGEN_ID_MSK); + /// <summary> + /// Invoked when the indexer is is called to + /// </summary> + /// <param name="key">The key/index to get the value for</param> + /// <returns>The value stored at the specified key</returns> + protected abstract string IndexerGet(string key); + /// <summary> + /// Sets a value requested by the indexer + /// </summary> + /// <param name="key">The key to associate the value with</param> + /// <param name="value">The value to store</param> + protected abstract void IndexerSet(string key, string value); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Sessions/SessionCacheLimitException.cs b/Plugins.Essentials/src/Sessions/SessionCacheLimitException.cs new file mode 100644 index 0000000..ffa4d9a --- /dev/null +++ b/Plugins.Essentials/src/Sessions/SessionCacheLimitException.cs @@ -0,0 +1,41 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: SessionCacheLimitException.cs +* +* SessionCacheLimitException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Plugins.Essentials.Sessions +{ + /// <summary> + /// Raised when the maximum number of cache entires has been reached, and the new session cannot be processed + /// </summary> + internal class SessionCacheLimitException : SessionException + { + public SessionCacheLimitException(string message) : base(message) + {} + public SessionCacheLimitException(string message, Exception innerException) : base(message, innerException) + {} + public SessionCacheLimitException() + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Sessions/SessionException.cs b/Plugins.Essentials/src/Sessions/SessionException.cs new file mode 100644 index 0000000..554c55f --- /dev/null +++ b/Plugins.Essentials/src/Sessions/SessionException.cs @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: SessionException.cs +* +* SessionException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Plugins.Essentials.Sessions +{ + /// <summary> + /// A base class for all session exceptions + /// </summary> + public class SessionException : Exception + { + ///<inheritdoc/> + public SessionException() + {} + ///<inheritdoc/> + public SessionException(string message) : base(message) + {} + ///<inheritdoc/> + public SessionException(string message, Exception innerException) : base(message, innerException) + {} + ///<inheritdoc/> + protected SessionException(SerializationInfo info, StreamingContext context) : base(info, context) + {} + } +} diff --git a/Plugins.Essentials/src/Sessions/SessionHandle.cs b/Plugins.Essentials/src/Sessions/SessionHandle.cs new file mode 100644 index 0000000..15c2743 --- /dev/null +++ b/Plugins.Essentials/src/Sessions/SessionHandle.cs @@ -0,0 +1,123 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: SessionHandle.cs +* +* SessionHandle.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading.Tasks; +using System.Diagnostics.CodeAnalysis; + +using VNLib.Net.Http; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Sessions +{ + public delegate ValueTask SessionReleaseCallback(ISession session, IHttpEvent @event); + + /// <summary> + /// A handle that holds exclusive access to a <see cref="ISession"/> + /// session object + /// </summary> + public readonly struct SessionHandle : IEquatable<SessionHandle> + { + /// <summary> + /// An empty <see cref="SessionHandle"/> instance. (A handle without a session object) + /// </summary> + public static readonly SessionHandle Empty = new(null, FileProcessArgs.Continue, null); + + private readonly SessionReleaseCallback? ReleaseCb; + + internal readonly bool IsSet => SessionData != null; + + /// <summary> + /// The session data object associated with the current session + /// </summary> + public readonly ISession? SessionData { get; } + + /// <summary> + /// A value indicating if the connection is valid and should continue to be processed + /// </summary> + public readonly FileProcessArgs EntityStatus { get; } + + /// <summary> + /// Initializes a new <see cref="SessionHandle"/> + /// </summary> + /// <param name="sessionData">The session data instance</param> + /// <param name="callback">A callback that is invoked when the handle is released</param> + /// <param name="entityStatus"></param> + public SessionHandle(ISession? sessionData, FileProcessArgs entityStatus, SessionReleaseCallback? callback) + { + SessionData = sessionData; + ReleaseCb = callback; + EntityStatus = entityStatus; + } + /// <summary> + /// Initializes a new <see cref="SessionHandle"/> + /// </summary> + /// <param name="sessionData">The session data instance</param> + /// <param name="callback">A callback that is invoked when the handle is released</param> + public SessionHandle(ISession sessionData, SessionReleaseCallback callback):this(sessionData, FileProcessArgs.Continue, callback) + {} + + /// <summary> + /// Releases the session from use + /// </summary> + /// <param name="event">The current connection event object</param> + public ValueTask ReleaseAsync(IHttpEvent @event) => ReleaseCb?.Invoke(SessionData!, @event) ?? ValueTask.CompletedTask; + + /// <summary> + /// Determines if another <see cref="SessionHandle"/> is equal to the current handle. + /// Handles are equal if neither handle is set or if their SessionData object is equal. + /// </summary> + /// <param name="other">The other handle to</param> + /// <returns>true if neither handle is set or if their SessionData object is equal, false otherwise</returns> + public bool Equals(SessionHandle other) + { + //If neither handle is set, then they are equal, otherwise they are equal if the session objects themselves are equal + return (!IsSet && !other.IsSet) || (SessionData?.Equals(other.SessionData) ?? false); + } + ///<inheritdoc/> + public override bool Equals([NotNullWhen(true)] object? obj) => (obj is SessionHandle other) && Equals(other); + ///<inheritdoc/> + public override int GetHashCode() + { + return IsSet ? SessionData!.GetHashCode() : base.GetHashCode(); + } + + /// <summary> + /// Checks if two <see cref="SessionHandle"/> instances are equal + /// </summary> + /// <param name="left"></param> + /// <param name="right"></param> + /// <returns></returns> + public static bool operator ==(SessionHandle left, SessionHandle right) => left.Equals(right); + + /// <summary> + /// Checks if two <see cref="SessionHandle"/> instances are not equal + /// </summary> + /// <param name="left"></param> + /// <param name="right"></param> + /// <returns></returns> + public static bool operator !=(SessionHandle left, SessionHandle right) => !(left == right); + } +} diff --git a/Plugins.Essentials/src/Sessions/SessionInfo.cs b/Plugins.Essentials/src/Sessions/SessionInfo.cs new file mode 100644 index 0000000..13e2a84 --- /dev/null +++ b/Plugins.Essentials/src/Sessions/SessionInfo.cs @@ -0,0 +1,231 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: SessionInfo.cs +* +* SessionInfo.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Net; +using System.Security.Authentication; +using System.Runtime.CompilerServices; + +using VNLib.Utils; +using VNLib.Net.Http; +using VNLib.Utils.Extensions; +using static VNLib.Plugins.Essentials.Statics; + +/* + * SessionInfo is a structure since it is only meant used in + * an HttpEntity context, so it may be allocated as part of + * the HttpEntity object, so have a single larger object + * passed by ref, and created once per request. It may even + * be cached and reused in the future. But for now user-apis + * should not be cached until a safe use policy is created. + */ + +#pragma warning disable CA1051 // Do not declare visible instance fields + +namespace VNLib.Plugins.Essentials.Sessions +{ + /// <summary> + /// When attached to a connection, provides persistant session storage and inforamtion based + /// on a connection. + /// </summary> + public readonly struct SessionInfo : IObjectStorage, IEquatable<SessionInfo> + { + /// <summary> + /// A value indicating if the current instance has been initiailzed + /// with a session. Otherwise properties are undefied + /// </summary> + public readonly bool IsSet; + + private readonly ISession UserSession; + /// <summary> + /// Key that identifies the current session. (Identical to cookie::sessionid) + /// </summary> + public readonly string SessionID; + /// <summary> + /// Session stored User-Agent + /// </summary> + public readonly string UserAgent; + /// <summary> + /// If the stored IP and current user's IP matches + /// </summary> + public readonly bool IPMatch; + /// <summary> + /// If the current connection and stored session have matching cross origin domains + /// </summary> + public readonly bool CrossOriginMatch; + /// <summary> + /// Flags the session as invalid. IMPORTANT: the user's session data is no longer valid and will throw an exception when accessed + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Invalidate(bool all = false) => UserSession.Invalidate(all); + /// <summary> + /// Marks the session ID to be regenerated during closing event + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void RegenID() => UserSession.RegenID(); + ///<inheritdoc/> + public T GetObject<T>(string key) + { + //Attempt to deserialze the object, or return default if it is empty + return this[key].AsJsonObject<T>(SR_OPTIONS); + } + ///<inheritdoc/> + public void SetObject<T>(string key, T obj) + { + //Serialize and store the object, or set null (remove) if the object is null + this[key] = obj?.ToJsonString(SR_OPTIONS); + } + + /// <summary> + /// Was the original session cross origin? + /// </summary> + public readonly bool CrossOrigin; + /// <summary> + /// The origin header specified during session creation + /// </summary> + public readonly Uri SpecifiedOrigin; + /// <summary> + /// Privilages associated with user specified during login + /// </summary> + public readonly DateTimeOffset Created; + /// <summary> + /// Was this session just created on this connection? + /// </summary> + public readonly bool IsNew; + /// <summary> + /// Gets or sets the session's login hash, if set to a non-empty/null value, will trigger an upgrade on close + /// </summary> + public readonly string LoginHash + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => UserSession.GetLoginToken(); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => UserSession.SetLoginToken(value); + } + /// <summary> + /// Gets or sets the session's login token, if set to a non-empty/null value, will trigger an upgrade on close + /// </summary> + public readonly string Token + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => UserSession.Token; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => UserSession.Token = value; + } + /// <summary> + /// <para> + /// Gets or sets the user-id for the current session. + /// </para> + /// <para> + /// Login code usually sets this value and it should be read-only + /// </para> + /// </summary> + public readonly string UserID + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => UserSession.UserID; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => UserSession.UserID = value; + } + /// <summary> + /// Privilages associated with user specified during login + /// </summary> + public readonly ulong Privilages + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => UserSession.Privilages; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => UserSession.Privilages = value; + } + /// <summary> + /// The IP address belonging to the client + /// </summary> + public readonly IPAddress UserIP; + /// <summary> + /// Was the session Initialy established on a secure connection? + /// </summary> + public readonly SslProtocols SecurityProcol; + /// <summary> + /// A value specifying the type of the backing session + /// </summary> + public readonly SessionType SessionType => UserSession.SessionType; + + /// <summary> + /// Accesses the session's general storage + /// </summary> + /// <param name="index">Key for specifie data</param> + /// <returns>Value associated with the key from the session's general storage</returns> + public readonly string this[string index] + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => UserSession[index]; + [MethodImpl(MethodImplOptions.AggressiveInlining)] + set => UserSession[index] = value; + } + + internal SessionInfo(ISession session, IConnectionInfo ci, IPAddress trueIp) + { + UserSession = session; + //Calculate and store + IsNew = session.IsNew; + SessionID = session.SessionID; + Created = session.Created; + UserIP = session.UserIP; + //Ip match + IPMatch = trueIp.Equals(session.UserIP); + //If the session is new, we can store intial security variables + if (session.IsNew) + { + session.InitNewSession(ci); + //Since all values will be the same as the connection, cache the connection values + UserAgent = ci.UserAgent; + SpecifiedOrigin = ci.Origin; + CrossOrigin = ci.CrossOrigin; + SecurityProcol = ci.SecurityProtocol; + } + else + { + //Load/decode stored variables + UserAgent = session.GetUserAgent(); + SpecifiedOrigin = session.GetOriginUri(); + CrossOrigin = session.IsCrossOrigin(); + SecurityProcol = session.GetSecurityProtocol(); + } + CrossOriginMatch = ci.Origin != null && ci.Origin.Equals(SpecifiedOrigin); + IsSet = true; + } + + ///<inheritdoc/> + public bool Equals(SessionInfo other) => SessionID.Equals(other.SessionID, StringComparison.Ordinal); + ///<inheritdoc/> + public override bool Equals(object obj) => obj is SessionInfo si && Equals(si); + ///<inheritdoc/> + public override int GetHashCode() => SessionID.GetHashCode(StringComparison.Ordinal); + ///<inheritdoc/> + public static bool operator ==(SessionInfo left, SessionInfo right) => left.Equals(right); + ///<inheritdoc/> + public static bool operator !=(SessionInfo left, SessionInfo right) => !(left == right); + + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Statics.cs b/Plugins.Essentials/src/Statics.cs new file mode 100644 index 0000000..58b5dd7 --- /dev/null +++ b/Plugins.Essentials/src/Statics.cs @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: Statics.cs +* +* Statics.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace VNLib.Plugins.Essentials +{ + public static class Statics + { + public static readonly JsonSerializerOptions SR_OPTIONS = new() + { + DictionaryKeyPolicy = JsonNamingPolicy.CamelCase, + NumberHandling = JsonNumberHandling.Strict, + ReadCommentHandling = JsonCommentHandling.Disallow, + WriteIndented = false, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + IgnoreReadOnlyFields = true, + DefaultBufferSize = Environment.SystemPageSize, + }; + } +} diff --git a/Plugins.Essentials/src/TimestampedCounter.cs b/Plugins.Essentials/src/TimestampedCounter.cs new file mode 100644 index 0000000..19cb8ec --- /dev/null +++ b/Plugins.Essentials/src/TimestampedCounter.cs @@ -0,0 +1,117 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: TimestampedCounter.cs +* +* TimestampedCounter.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; + + +#nullable enable + +namespace VNLib.Plugins.Essentials +{ + /// <summary> + /// Stucture that allows for convient storage of a counter value + /// and a second precision timestamp into a 64-bit unsigned integer + /// </summary> + public readonly struct TimestampedCounter : IEquatable<TimestampedCounter> + { + /// <summary> + /// The time the count was last modifed + /// </summary> + public readonly DateTimeOffset LastModified; + /// <summary> + /// The last failed login attempt count value + /// </summary> + public readonly uint Count; + + /// <summary> + /// Initalizes a new flc structure with the current UTC date + /// and the specified count value + /// </summary> + /// <param name="count">FLC current count</param> + public TimestampedCounter(uint count) : this(DateTimeOffset.UtcNow, count) + { } + + private TimestampedCounter(DateTimeOffset dto, uint count) + { + Count = count; + LastModified = dto; + } + + /// <summary> + /// Compacts and converts the counter value and timestamp into + /// a 64bit unsigned integer + /// </summary> + /// <param name="count">The counter to convert</param> + public static explicit operator ulong(TimestampedCounter count) => count.ToUInt64(); + + /// <summary> + /// Compacts and converts the counter value and timestamp into + /// a 64bit unsigned integer + /// </summary> + /// <returns>The uint64 compacted value</returns> + public ulong ToUInt64() + { + //Upper 32 bits time, lower 32 bits count + ulong value = (ulong)LastModified.ToUnixTimeSeconds() << 32; + value |= Count; + return value; + } + + /// <summary> + /// The previously compacted <see cref="TimestampedCounter"/> + /// value to cast back to a counter + /// </summary> + /// <param name="value"></param> + public static explicit operator TimestampedCounter(ulong value) => FromUInt64(value); + + ///<inheritdoc/> + public override bool Equals(object? obj) => obj is TimestampedCounter counter && Equals(counter); + ///<inheritdoc/> + public override int GetHashCode() => this.ToUInt64().GetHashCode(); + ///<inheritdoc/> + public static bool operator ==(TimestampedCounter left, TimestampedCounter right) => left.Equals(right); + ///<inheritdoc/> + public static bool operator !=(TimestampedCounter left, TimestampedCounter right) => !(left == right); + ///<inheritdoc/> + public bool Equals(TimestampedCounter other) => ToUInt64() == other.ToUInt64(); + + /// <summary> + /// The previously compacted <see cref="TimestampedCounter"/> + /// value to cast back to a counter + /// </summary> + /// <param name="value">The uint64 encoded <see cref="TimestampedCounter"/></param> + /// <returns> + /// The decoded <see cref="TimestampedCounter"/> from its + /// compatcted representation + /// </returns> + public static TimestampedCounter FromUInt64(ulong value) + { + //Upper 32 bits time, lower 32 bits count + long time = (long)(value >> 32); + uint count = (uint)(value & uint.MaxValue); + //Init dto struct + return new(DateTimeOffset.FromUnixTimeSeconds(time), count); + } + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/IUser.cs b/Plugins.Essentials/src/Users/IUser.cs new file mode 100644 index 0000000..28c5305 --- /dev/null +++ b/Plugins.Essentials/src/Users/IUser.cs @@ -0,0 +1,74 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: IUser.cs +* +* IUser.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; + +using VNLib.Utils; +using VNLib.Utils.Async; +using VNLib.Utils.Memory; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Users +{ + /// <summary> + /// Represents an abstract user account + /// </summary> + public interface IUser : IAsyncExclusiveResource, IDisposable, IObjectStorage, IEnumerable<KeyValuePair<string, string>>, IIndexable<string, string> + { + /// <summary> + /// The user's privilage level + /// </summary> + ulong Privilages { get; } + /// <summary> + /// The user's ID + /// </summary> + string UserID { get; } + /// <summary> + /// Date the user's account was created + /// </summary> + DateTimeOffset Created { get; } + /// <summary> + /// The user's password hash if retreived from the backing store, otherwise null + /// </summary> + PrivateString? PassHash { get; } + /// <summary> + /// Status of account + /// </summary> + UserStatus Status { get; set; } + /// <summary> + /// Is the account only usable from local network? + /// </summary> + bool LocalOnly { get; set; } + /// <summary> + /// The user's email address + /// </summary> + string EmailAddress { get; set; } + /// <summary> + /// Marks the user for deletion on release + /// </summary> + void Delete(); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/IUserManager.cs b/Plugins.Essentials/src/Users/IUserManager.cs new file mode 100644 index 0000000..dd521e4 --- /dev/null +++ b/Plugins.Essentials/src/Users/IUserManager.cs @@ -0,0 +1,103 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: IUserManager.cs +* +* IUserManager.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils; +using VNLib.Utils.Memory; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Users +{ + /// <summary> + /// A backing store that provides user accounts + /// </summary> + public interface IUserManager + { + /// <summary> + /// Attempts to get a user object without their password from the database asynchronously + /// </summary> + /// <param name="userId">The id of the user</param> + /// <param name="cancellationToken">A token to cancel the operation</param> + /// <returns>The user's <see cref="IUser"/> object, null if the user was not found</returns> + /// <exception cref="ArgumentNullException"></exception> + Task<IUser?> GetUserFromIDAsync(string userId, CancellationToken cancellationToken = default); + /// <summary> + /// Attempts to get a user object without their password from the database asynchronously + /// </summary> + /// <param name="emailAddress">The user's email address</param> + /// <param name="cancellationToken">A token to cancel the operation</param> + /// <returns>The user's <see cref="IUser"/> object, null if the user was not found</returns> + /// <exception cref="ArgumentNullException"></exception> + Task<IUser?> GetUserFromEmailAsync(string emailAddress, CancellationToken cancellationToken = default); + /// <summary> + /// Attempts to get a user object with their password from the database on the current thread + /// </summary> + /// <param name="userid">The id of the user</param> + /// <param name="cancellation">A token to cancel the operation</param> + /// <returns>The user's <see cref="IUser"/> object, null if the user was not found</returns> + /// <exception cref="ArgumentNullException"></exception> + Task<IUser?> GetUserAndPassFromIDAsync(string userid, CancellationToken cancellation = default); + /// <summary> + /// Attempts to get a user object with their password from the database asynchronously + /// </summary> + /// <param name="emailAddress">The user's email address</param> + /// <param name="cancellationToken">A token to cancel the operation</param> + /// <returns>The user's <see cref="IUser"/> object, null if the user was not found</returns> + /// <exception cref="ArgumentNullException"></exception> + Task<IUser?> GetUserAndPassFromEmailAsync(string emailAddress, CancellationToken cancellationToken = default); + /// <summary> + /// Creates a new user in the current user's table and if successful returns the new user object (without password) + /// </summary> + /// <param name="userid">The user id</param> + /// <param name="privilages">A number representing the privilage level of the account</param> + /// <param name="passHash">Value to store in the password field</param> + /// <param name="cancellation">A token to cancel the operation</param> + /// <param name="emailAddress">The account email address</param> + /// <returns>An object representing a user's account if successful, null otherwise</returns> + /// <exception cref="UserExistsException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="UserCreationFailedException"></exception> + Task<IUser> CreateUserAsync(string userid, string emailAddress, ulong privilages, PrivateString passHash, CancellationToken cancellation = default); + /// <summary> + /// Updates a password associated with the specified user. If the update fails, the transaction + /// is rolled back. + /// </summary> + /// <param name="user">The user account to update the password of</param> + /// <param name="newPass">The new password to set</param> + /// <param name="cancellation">A token to cancel the operation</param> + /// <returns>The result of the operation, the result should be 1 (aka true)</returns> + Task<ERRNO> UpdatePassAsync(IUser user, PrivateString newPass, CancellationToken cancellation = default); + + /// <summary> + /// Gets the number of entries in the current user table + /// </summary> + /// <param name="cancellation">A token to cancel the operation</param> + /// <returns>The number of users in the table, or -1 if the operation failed</returns> + Task<long> GetUserCountAsync(CancellationToken cancellation = default); + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/UserCreationFailedException.cs b/Plugins.Essentials/src/Users/UserCreationFailedException.cs new file mode 100644 index 0000000..9f509ac --- /dev/null +++ b/Plugins.Essentials/src/Users/UserCreationFailedException.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UserCreationFailedException.cs +* +* UserCreationFailedException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; +using VNLib.Utils.Resources; + +#nullable enable + +namespace VNLib.Plugins.Essentials.Users +{ + /// <summary> + /// Raised when a user creation operation has failed and could not be created + /// </summary> + public class UserCreationFailedException : ResourceUpdateFailedException + { + public UserCreationFailedException() + {} + public UserCreationFailedException(string message) : base(message) + {} + public UserCreationFailedException(string message, Exception innerException) : base(message, innerException) + {} + protected UserCreationFailedException(SerializationInfo info, StreamingContext context) : base(info, context) + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/UserDeleteException.cs b/Plugins.Essentials/src/Users/UserDeleteException.cs new file mode 100644 index 0000000..cd26543 --- /dev/null +++ b/Plugins.Essentials/src/Users/UserDeleteException.cs @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UserDeleteException.cs +* +* UserDeleteException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using VNLib.Utils.Resources; + +namespace VNLib.Plugins.Essentials.Users +{ + /// <summary> + /// Raised when a user flagged for deletion could not be deleted. See the <see cref="Exception.InnerException"/> + /// for the Exception that cause the opertion to fail + /// </summary> + public class UserDeleteException : ResourceDeleteFailedException + { + public UserDeleteException(string message, Exception cause) : base(message, cause) { } + + public UserDeleteException() + {} + + public UserDeleteException(string message) : base(message) + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/UserExistsException.cs b/Plugins.Essentials/src/Users/UserExistsException.cs new file mode 100644 index 0000000..5c63547 --- /dev/null +++ b/Plugins.Essentials/src/Users/UserExistsException.cs @@ -0,0 +1,49 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UserExistsException.cs +* +* UserExistsException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Plugins.Essentials.Users +{ + /// <summary> + /// Raised when an <see cref="IUserManager"/> operation + /// fails because the user account already exists + /// </summary> + public class UserExistsException : UserCreationFailedException + { + ///<inheritdoc/> + public UserExistsException() + {} + ///<inheritdoc/> + public UserExistsException(string message) : base(message) + {} + ///<inheritdoc/> + public UserExistsException(string message, Exception innerException) : base(message, innerException) + {} + ///<inheritdoc/> + protected UserExistsException(SerializationInfo info, StreamingContext context) : base(info, context) + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/UserStatus.cs b/Plugins.Essentials/src/Users/UserStatus.cs new file mode 100644 index 0000000..32aa63d --- /dev/null +++ b/Plugins.Essentials/src/Users/UserStatus.cs @@ -0,0 +1,50 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UserStatus.cs +* +* UserStatus.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +namespace VNLib.Plugins.Essentials.Users +{ + public enum UserStatus + { + /// <summary> + /// Unverified account state + /// </summary> + Unverified, + /// <summary> + /// Active account state. The account is fully functional + /// </summary> + Active, + /// <summary> + /// The account is suspended + /// </summary> + Suspended, + /// <summary> + /// The account is inactive as marked by the system + /// </summary> + Inactive, + /// <summary> + /// The account has been locked from access + /// </summary> + Locked + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/Users/UserUpdateException.cs b/Plugins.Essentials/src/Users/UserUpdateException.cs new file mode 100644 index 0000000..391bb05 --- /dev/null +++ b/Plugins.Essentials/src/Users/UserUpdateException.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: UserUpdateException.cs +* +* UserUpdateException.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using VNLib.Utils.Resources; + +namespace VNLib.Plugins.Essentials.Users +{ + /// <summary> + /// Raised when a user-data object was modified and an update operation failed + /// </summary> + public class UserUpdateException : ResourceUpdateFailedException + { + public UserUpdateException(string message, Exception cause) : base(message, cause) { } + + public UserUpdateException() + {} + + public UserUpdateException(string message) : base(message) + {} + } +}
\ No newline at end of file diff --git a/Plugins.Essentials/src/VNLib.Plugins.Essentials.csproj b/Plugins.Essentials/src/VNLib.Plugins.Essentials.csproj new file mode 100644 index 0000000..b1751bd --- /dev/null +++ b/Plugins.Essentials/src/VNLib.Plugins.Essentials.csproj @@ -0,0 +1,53 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <RootNamespace>VNLib.Plugins.Essentials</RootNamespace> + <Company>$(Authors)</Company> + <Authors>Vaughn Nugent</Authors> + <Product>VNLib Essentials Plugin Library</Product> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <AssemblyVersion></AssemblyVersion> + <FileVersion></FileVersion> + <Description>Provides essential web, user, storage, and database interaction features for use with web applications</Description> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <AssemblyName>VNLib.Plugins.Essentials</AssemblyName> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <!-- Resolve nuget dll files and store them in the output dir --> + <PropertyGroup> + <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> + <PackageTags>VNLib, Plugins, VNLib.Plugins.Essentials, Essentials, Essential Plugins, HTTP Essentials, OAuth2</PackageTags> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <Version>1.0.1.3</Version> + <AnalysisLevel>latest-all</AnalysisLevel> + <EnableNETAnalyzers>True</EnableNETAnalyzers> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Hashing\src\VNLib.Hashing.Portable.csproj" /> + <ProjectReference Include="..\..\Http\src\VNLib.Net.Http.csproj" /> + <ProjectReference Include="..\..\Plugins\src\VNLib.Plugins.csproj" /> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + </ItemGroup> + +</Project> diff --git a/Plugins.Essentials/src/WebSocketSession.cs b/Plugins.Essentials/src/WebSocketSession.cs new file mode 100644 index 0000000..106501c --- /dev/null +++ b/Plugins.Essentials/src/WebSocketSession.cs @@ -0,0 +1,204 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Essentials +* File: WebSocketSession.cs +* +* WebSocketSession.cs is part of VNLib.Plugins.Essentials which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Essentials is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as +* published by the Free Software Foundation, either version 3 of the +* License, or (at your option) any later version. +* +* VNLib.Plugins.Essentials is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with this program. If not, see https://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Net.WebSockets; +using System.Threading.Tasks; + +using VNLib.Net.Http; + +#nullable enable + +namespace VNLib.Plugins.Essentials +{ + /// <summary> + /// A callback method to invoke when an HTTP service successfully transfers protocols to + /// the WebSocket protocol and the socket is ready to be used + /// </summary> + /// <param name="session">The open websocket session instance</param> + /// <returns> + /// A <see cref="Task"/> that will be awaited by the HTTP layer. When the task completes, the transport + /// will be closed and the session disposed + /// </returns> + + public delegate Task WebsocketAcceptedCallback(WebSocketSession session); + + /// <summary> + /// Represents a <see cref="WebSocket"/> wrapper to manage the lifetime of the captured + /// connection context and the underlying transport. This session is managed by the parent + /// <see cref="HttpServer"/> that it was created on. + /// </summary> + public sealed class WebSocketSession : AlternateProtocolBase + { + private WebSocket? WsHandle; + private readonly WebsocketAcceptedCallback AcceptedCallback; + + /// <summary> + /// A cancellation token that can be monitored to reflect the state + /// of the webscocket + /// </summary> + public CancellationToken Token => CancelSource.Token; + + /// <summary> + /// Id assigned to this instance on creation + /// </summary> + public string SocketID { get; } + + /// <summary> + /// Negotiated sub-protocol + /// </summary> + public string? SubProtocol { get; } + + /// <summary> + /// A user-defined state object passed during socket accept handshake + /// </summary> + public object? UserState { get; internal set; } + + internal WebSocketSession(string? subProtocol, WebsocketAcceptedCallback callback) + : this(Guid.NewGuid().ToString("N"), subProtocol, callback) + { } + + internal WebSocketSession(string socketId, string? subProtocol, WebsocketAcceptedCallback callback) + { + SocketID = socketId; + SubProtocol = subProtocol; + //Store the callback function + AcceptedCallback = callback; + } + + /// <summary> + /// Initialzes the created websocket with the specified protocol + /// </summary> + /// <param name="transport">Transport stream to use for the websocket</param> + /// <returns>The accept callback function specified during object initialization</returns> + protected override async Task RunAsync(Stream transport) + { + try + { + WebSocketCreationOptions ce = new() + { + IsServer = true, + KeepAliveInterval = TimeSpan.FromSeconds(30), + SubProtocol = SubProtocol, + }; + + //Create a new websocket from the context stream + WsHandle = WebSocket.CreateFromStream(transport, ce); + + //Register token to abort the websocket so the managed ws uses the non-fallback send/recv method + using CancellationTokenRegistration abortReg = Token.Register(WsHandle.Abort); + + //Return the callback function to explcitly invoke it + await AcceptedCallback(this); + } + finally + { + WsHandle?.Dispose(); + UserState = null; + } + } + + /// <summary> + /// Asynchronously receives data from the Websocket and copies the data to the specified buffer + /// </summary> + /// <param name="buffer">The buffer to store read data</param> + /// <returns>A task that resolves a <see cref="WebSocketReceiveResult"/> which contains the status of the operation</returns> + /// <exception cref="OperationCanceledException"></exception> + public Task<WebSocketReceiveResult> ReceiveAsync(ArraySegment<byte> buffer) + { + //Begin receive operation only with the internal token + return WsHandle!.ReceiveAsync(buffer, CancellationToken.None); + } + + /// <summary> + /// Asynchronously receives data from the Websocket and copies the data to the specified buffer + /// </summary> + /// <param name="buffer">The buffer to store read data</param> + /// <returns></returns> + /// <exception cref="OperationCanceledException"></exception> + public ValueTask<ValueWebSocketReceiveResult> ReceiveAsync(Memory<byte> buffer) + { + //Begin receive operation only with the internal token + return WsHandle!.ReceiveAsync(buffer, CancellationToken.None); + } + + /// <summary> + /// Asynchronously sends the specified buffer to the client of the specified type + /// </summary> + /// <param name="buffer">The buffer containing data to send</param> + /// <param name="type">The message/data type of the packet to send</param> + /// <param name="endOfMessage">A value that indicates this message is the final message of the transaction</param> + /// <returns></returns> + /// <exception cref="OperationCanceledException"></exception> + public Task SendAsync(ArraySegment<byte> buffer, WebSocketMessageType type, bool endOfMessage) + { + //Create a send request with + return WsHandle!.SendAsync(buffer, type, endOfMessage, CancellationToken.None); + } + + /// <summary> + /// Asynchronously sends the specified buffer to the client of the specified type + /// </summary> + /// <param name="buffer">The buffer containing data to send</param> + /// <param name="type">The message/data type of the packet to send</param> + /// <param name="endOfMessage">A value that indicates this message is the final message of the transaction</param> + /// <returns></returns> + /// <exception cref="OperationCanceledException"></exception> + public ValueTask SendAsync(ReadOnlyMemory<byte> buffer, WebSocketMessageType type, bool endOfMessage) + { + //Begin receive operation only with the internal token + return WsHandle!.SendAsync(buffer, type, endOfMessage, CancellationToken.None); + } + + + /// <summary> + /// Properly closes a currently connected websocket + /// </summary> + /// <param name="status">Set the close status</param> + /// <param name="reason">Set the close reason</param> + /// <exception cref="ObjectDisposedException"></exception> + public Task CloseSocketAsync(WebSocketCloseStatus status, string reason) + { + return WsHandle!.CloseAsync(status, reason, CancellationToken.None); + } + + /// <summary> + /// + /// </summary> + /// <param name="status"></param> + /// <param name="reason"></param> + /// <param name="cancellation"></param> + /// <returns></returns> + public Task CloseSocketOutputAsync(WebSocketCloseStatus status, string reason, CancellationToken cancellation = default) + { + if (WsHandle!.State == WebSocketState.Open || WsHandle.State == WebSocketState.CloseSent) + { + return WsHandle.CloseOutputAsync(status, reason, cancellation); + } + return Task.CompletedTask; + } + } +}
\ No newline at end of file diff --git a/Plugins.Runtime/LICENSE.txt b/Plugins.Runtime/LICENSE.txt new file mode 100644 index 0000000..2848520 --- /dev/null +++ b/Plugins.Runtime/LICENSE.txt @@ -0,0 +1,346 @@ +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). + +SPDX-License-Identifier: GPL-2.0-or-later + +License-Text: + +GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + VNLib.Hashing.Portable is a compact .NET managed cryptographic operation + utilities library. + Copyright (C) 2022 Vaughn Nugent + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License.
\ No newline at end of file diff --git a/Plugins.Runtime/README.md b/Plugins.Runtime/README.md new file mode 100644 index 0000000..1146c9b --- /dev/null +++ b/Plugins.Runtime/README.md @@ -0,0 +1,53 @@ +# VNLib.Plugins.Runtime + +A library that manages the runtime loading/unloading of a managed .NET assembly that exposes one or more types that implement the VNLib.Plugins.IPlugin interface, and the plugins lifecycle. The `DynamicPluginLoader` class also handles "hot" assembly reload and exposes lifecycle hooks for applications to correctly detect those changes. + +### 3rd Party Dependencies +This library does not, modify, contribute, or affect the functionality of any of the 3rd party libraries below. + +**VNLib.Plugins.Runtime** relies on a single 3rd party library [McMaster.NETCore.Plugins](https://github.com/natemcmaster/DotNetCorePlugins) from [Nate McMaster](https://github.com/natemcmaster) for runtime assembly loading. You must include this dependency in your project. + +## Usage +An XML documentation file is included for all public apis. + +```programming language C# + //RuntimePluginLoader is disposable to cleanup optional config files and cleanup all PluginLoader resources + using RuntimePluginLoader plugin = new(<fqAssemblyPath>,...<args>); + + //Load assembly, optional config file, capture all IPlugin types, then call IPlugin.Load() and all other lifecycle hooks + await plugin.InitLoaderAsync(); + //Listen for reload events + plugin.Reloaded += (object? loader, EventAargs = null) =>{}; + + //Get all endpoints from all exposed plugins + IEndpoint[] endpoints = plugin.GetEndpoints().ToArray(); + + //Your plugin types may also expose custom types, you may see if they are available + if(plugin.ExposesType<IMyCustomType>()) + { + IMyCustomType mt = plugin.GetExposedTypeFromPlugin<IMyCustomType>(); + } + + //Trigger manual reload, will unload, then reload and trigger events + plugin.ReloadPlugin(); + + //Unload all plugins + plugin.UnloadAll(); + + //Leaving scope disposes the loader +``` +### Warnings +##### Load/Unload/Hot reload +When hot-reload is disabled and manual reloading is not expected, or unloading is also disabled, you not worry about reload events since the assemblies will never be unloaded. If unloading is disabled and `RuntimePluginLoader.UnloadAll()` is called, only the IPlugin lifecycle hooks will be called (`IPlugin.Unload();`), internal collections are cleared, but no other actions take place. + +`RuntimePluginLoader.UnloadAll()` Should only be called when you are no longer using the assembly, and all **IPlugin** instances or custom types. The **VNLib.Plugins.Essentials.ServiceStack** library is careful to remove all instances of the exposed plugins, their endpoints, and all other custom types that were exposed, before calling this method. + +Disposing the **RuntimePluginLoader** does not unload the plugins, but simply disposes any internal resources disposes the internal **PluginLoader**, so it should only be disposed after `RuntimePluginLoader.UnloadAll()` is called. + +_Please see [McMaster.NETCore.Plugins](https://github.com/natemcmaster/DotNetCorePlugins) for more information on runtime .NET assembly loading and the dangers of doing so_ + +**Hot reload should only be enabled for debugging/development purposes, you should understand the security implications and compatibility of .NET collectable assemblies** + +## License +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). +See the LICENSE files for more information. diff --git a/Plugins.Runtime/src/LivePlugin.cs b/Plugins.Runtime/src/LivePlugin.cs new file mode 100644 index 0000000..0001990 --- /dev/null +++ b/Plugins.Runtime/src/LivePlugin.cs @@ -0,0 +1,220 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Runtime +* File: LivePlugin.cs +* +* LivePlugin.cs is part of VNLib.Plugins.Runtime which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Runtime is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins.Runtime is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins.Runtime. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Linq; +using System.Reflection; +using System.Text.Json; + +using VNLib.Utils.Logging; +using VNLib.Utils.Extensions; +using VNLib.Plugins.Attributes; + +namespace VNLib.Plugins.Runtime +{ + /// <summary> + /// <para> + /// Wrapper for a loaded <see cref="IPlugin"/> instance, used internally + /// for a single instance. + /// </para> + /// <para> + /// Lifetime: for the existance of a single loaded + /// plugin instance. Created once per loaded plugin instance. Once the plugin + /// is unloaded, it is no longer useable. + /// </para> + /// </summary> + public class LivePlugin : IEquatable<IPlugin>, IEquatable<LivePlugin> + { + /// <summary> + /// The plugin's <see cref="IPlugin.PluginName"/> property during load time + /// </summary> + /// <exception cref="InvalidOperationException"></exception> + public string PluginName => Plugin?.PluginName ?? throw new InvalidOperationException("Plugin is not loaded"); + + /// <summary> + /// The underlying <see cref="IPlugin"/> that is warpped + /// by he current instance + /// </summary> + public IPlugin? Plugin { get; private set; } + + private readonly Type PluginType; + + private ConsoleEventHandler? PluginConsoleHandler; + + internal LivePlugin(IPlugin plugin) + { + Plugin = plugin; + PluginType = plugin.GetType(); + GetConsoleHandler(); + } + + private void GetConsoleHandler() + { + //Get the console handler method from the plugin instance + MethodInfo? handler = (from m in PluginType.GetMethods() + where m.GetCustomAttribute<ConsoleEventHandlerAttribute>() != null + select m) + .FirstOrDefault(); + //Get a delegate handler for the plugin + PluginConsoleHandler = handler?.CreateDelegate<ConsoleEventHandler>(Plugin); + } + + /// <summary> + /// Sets the plugin's configuration if it defines a <see cref="ConfigurationInitalizerAttribute"/> + /// on an instance method + /// </summary> + /// <param name="hostConfig">The host configuration DOM</param> + /// <param name="pluginConf">The plugin local configuration DOM</param> + internal void InitConfig(JsonDocument hostConfig, JsonDocument pluginConf) + { + //Get the console handler method from the plugin instance + MethodInfo? confHan = PluginType.GetMethods().Where(static m => m.GetCustomAttribute<ConfigurationInitalizerAttribute>() != null) + .FirstOrDefault(); + //Get a delegate handler for the plugin + ConfigInitializer? configInit = confHan?.CreateDelegate<ConfigInitializer>(Plugin); + if (configInit == null) + { + return; + } + //Merge configurations before passing to plugin + JsonDocument merged = hostConfig.Merge(pluginConf, "host", PluginType.Name); + try + { + //Invoke + configInit.Invoke(merged); + } + catch + { + merged.Dispose(); + throw; + } + } + + /// <summary> + /// Invokes the plugin's log initalizer method if it defines a <see cref="LogInitializerAttribute"/> + /// on an instance method + /// </summary> + /// <param name="cliArgs">The current process's CLI args</param> + internal void InitLog(string[] cliArgs) + { + //Get the console handler method from the plugin instance + MethodInfo? logInit = (from m in PluginType.GetMethods() + where m.GetCustomAttribute<LogInitializerAttribute>() != null + select m) + .FirstOrDefault(); + //Get a delegate handler for the plugin + LogInitializer? logFunc = logInit?.CreateDelegate<LogInitializer>(Plugin); + //Invoke + logFunc?.Invoke(cliArgs); + } + + /// <summary> + /// Invokes the plugins console event handler if the type has one + /// and the plugin is loaded. + /// </summary> + /// <param name="message">The message to pass to the plugin handler</param> + /// <returns> + /// True if the command was sent to the plugin, false if the plugin is + /// unloaded or did not export a console event handler + /// </returns> + public bool SendConsoleMessage(string message) + { + //Make sure plugin is loaded and has a console handler + if (PluginConsoleHandler == null) + { + return false; + } + //Invoke plugin console handler + PluginConsoleHandler(message); + return true; + } + + /// <summary> + /// Calls the <see cref="IPlugin.Load"/> method on the plugin if its loaded + /// </summary> + internal void LoadPlugin() => Plugin?.Load(); + + /// <summary> + /// Unloads all loaded endpoints from + /// that they were loaded to, then unloads the plugin. + /// </summary> + /// <param name="logSink">An optional log provider to write unload exceptions to</param> + /// <remarks> + /// If <paramref name="logSink"/> is no null unload exceptions are swallowed and written to the log + /// </remarks> + internal void UnloadPlugin(ILogProvider? logSink) + { + /* + * We need to swallow plugin unload errors to avoid + * unknown state, making sure endpoints are properly + * unloaded! + */ + try + { + //Unload the plugin + Plugin?.Unload(); + } + catch (Exception ex) + { + //Create an unload wrapper for the exception + PluginUnloadException wrapper = new("Exception raised during plugin unload", ex); + if (logSink == null) + { + throw wrapper; + } + //Write error to log sink + logSink.Error(wrapper); + } + Plugin = null; + PluginConsoleHandler = null; + } + ///<inheritdoc/> + public override bool Equals(object? obj) + { + Type? pluginType = Plugin?.GetType(); + Type? otherType = obj?.GetType(); + if(pluginType == null || otherType == null) + { + return false; + } + //If the other plugin is the same type as the current instance return true + return pluginType.FullName == otherType.FullName; + } + ///<inheritdoc/> + public bool Equals(LivePlugin? other) + { + return Equals(other?.Plugin); + } + ///<inheritdoc/> + public bool Equals(IPlugin? other) + { + return Equals((object?)other); + } + ///<inheritdoc/> + public override int GetHashCode() + { + return Plugin?.GetHashCode() ?? throw new InvalidOperationException("Plugin is null"); + } + } +} diff --git a/Plugins.Runtime/src/LoaderExtensions.cs b/Plugins.Runtime/src/LoaderExtensions.cs new file mode 100644 index 0000000..795dcf5 --- /dev/null +++ b/Plugins.Runtime/src/LoaderExtensions.cs @@ -0,0 +1,120 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Runtime +* File: LoaderExtensions.cs +* +* LoaderExtensions.cs is part of VNLib.Plugins.Runtime which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Runtime is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins.Runtime is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins.Runtime. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; +using System.Linq; + +namespace VNLib.Plugins.Runtime +{ + public static class LoaderExtensions + { + /// <summary> + /// Searches all plugins within the current loader for a + /// single plugin that derrives the specified type + /// </summary> + /// <typeparam name="T">The type the plugin must derrive from</typeparam> + /// <param name="loader"></param> + /// <returns>The instance of the plugin that derrives from the specified type</returns> + public static LivePlugin? GetExposedPlugin<T>(this RuntimePluginLoader loader) + { + return loader.LivePlugins + .Where(static pl => typeof(T).IsAssignableFrom(pl.Plugin!.GetType())) + .SingleOrDefault(); + } + + /// <summary> + /// Searches all plugins within the current loader for a + /// single plugin that derrives the specified type + /// </summary> + /// <typeparam name="T">The type the plugin must derrive from</typeparam> + /// <param name="loader"></param> + /// <returns>The instance of your custom type casted, or null if not found or could not be casted</returns> + public static T? GetExposedTypeFromPlugin<T>(this RuntimePluginLoader loader) where T: class + { + LivePlugin? plugin = loader.LivePlugins + .Where(static pl => typeof(T).IsAssignableFrom(pl.Plugin!.GetType())) + .SingleOrDefault(); + + return plugin?.Plugin as T; + } + + /// <summary> + /// Registers a listener delegate method to invoke when the + /// current <see cref="RuntimePluginLoader"/> is reloaded, and passes + /// the new instance of the specified type + /// </summary> + /// <typeparam name="T">The single plugin type to register a listener for</typeparam> + /// <param name="loader"></param> + /// <param name="reloaded">The delegate method to invoke when the loader has reloaded plugins</param> + /// <exception cref="ArgumentNullException"></exception> + public static bool RegisterListenerForSingle<T>(this RuntimePluginLoader loader, Action<T, T> reloaded) where T: class + { + _ = reloaded ?? throw new ArgumentNullException(nameof(reloaded)); + + //try to get the casted type from the loader + T? current = loader.GetExposedTypeFromPlugin<T>(); + + if (current == null) + { + return false; + } + else + { + loader.Reloaded += delegate (object? sender, EventArgs args) + { + RuntimePluginLoader wpl = (sender as RuntimePluginLoader)!; + //Get the new loaded type + T newT = (wpl.GetExposedPlugin<T>()!.Plugin as T)!; + //Invoke reloaded action + reloaded(current, newT); + //update the new current instance + current = newT; + }; + + return true; + } + } + + /// <summary> + /// Gets all endpoints exposed by all exported plugin instances + /// within the current loader + /// </summary> + /// <param name="loader"></param> + /// <returns>An enumeration of all endpoints</returns> + public static IEnumerable<IEndpoint> GetEndpoints(this RuntimePluginLoader loader) => loader.LivePlugins.SelectMany(static pl => pl.Plugin!.GetEndpoints()); + + /// <summary> + /// Determines if any loaded plugin types exposes an instance of the + /// specified type + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="loader"></param> + /// <returns>True if any plugin instance exposes a the specified type, false otherwise</returns> + public static bool ExposesType<T>(this RuntimePluginLoader loader) where T : class + { + return loader.LivePlugins.Any(static pl => typeof(T).IsAssignableFrom(pl.Plugin?.GetType())); + } + } +} diff --git a/Plugins.Runtime/src/PluginUnloadExcpetion.cs b/Plugins.Runtime/src/PluginUnloadExcpetion.cs new file mode 100644 index 0000000..53f63b2 --- /dev/null +++ b/Plugins.Runtime/src/PluginUnloadExcpetion.cs @@ -0,0 +1,46 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Runtime +* File: PluginUnloadExcpetion.cs +* +* PluginUnloadExcpetion.cs is part of VNLib.Plugins.Runtime which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Runtime is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins.Runtime is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins.Runtime. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Plugins.Runtime +{ + /// <summary> + /// A wrapper for exceptions that are raised during an + /// assembly plugin unload event. See <see cref="Exception.InnerException"/> + /// for details + /// </summary> + public class PluginUnloadException : Exception + { + public PluginUnloadException() + {} + public PluginUnloadException(string message) : base(message) + {} + public PluginUnloadException(string message, Exception innerException) : base(message, innerException) + {} + protected PluginUnloadException(SerializationInfo info, StreamingContext context) : base(info, context) + {} + } +} diff --git a/Plugins.Runtime/src/RuntimePluginLoader.cs b/Plugins.Runtime/src/RuntimePluginLoader.cs new file mode 100644 index 0000000..c688f8b --- /dev/null +++ b/Plugins.Runtime/src/RuntimePluginLoader.cs @@ -0,0 +1,250 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins.Runtime +* File: DynamicPluginLoader.cs +* +* DynamicPluginLoader.cs is part of VNLib.Plugins.Runtime which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins.Runtime is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins.Runtime is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins.Runtime. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Reflection; +using System.Runtime.Loader; +using System.Threading.Tasks; +using System.Collections.Generic; + +using McMaster.NETCore.Plugins; + +using VNLib.Utils; +using VNLib.Utils.IO; +using VNLib.Utils.Logging; +using VNLib.Utils.Extensions; + + +namespace VNLib.Plugins.Runtime +{ + /// <summary> + /// A runtime .NET assembly loader specialized to load + /// assemblies that export <see cref="IPlugin"/> types. + /// </summary> + public class RuntimePluginLoader : VnDisposeable + { + protected readonly PluginLoader Loader; + protected readonly string PluginPath; + protected readonly JsonDocument HostConfig; + protected readonly ILogProvider? Log; + protected readonly LinkedList<LivePlugin> LoadedPlugins; + + /// <summary> + /// A readonly collection of all loaded plugin wrappers + /// </summary> + public IReadOnlyCollection<LivePlugin> LivePlugins => LoadedPlugins; + + /// <summary> + /// An event that is raised before the loader + /// unloads all plugin instances + /// </summary> + protected event EventHandler<PluginReloadedEventArgs>? OnBeforeReloaded; + /// <summary> + /// An event that is raised after a successfull reload of all new + /// plugins for the instance + /// </summary> + protected event EventHandler? OnAfterReloaded; + + /// <summary> + /// Raised when the current loader has reloaded the assembly and + /// all plugins were successfully loaded. + /// </summary> + public event EventHandler? Reloaded; + + /// <summary> + /// The current plugin's JSON configuration DOM loaded from the plugin's directory + /// if it exists. Only valid after first initalization + /// </summary> + public JsonDocument? PluginConfigDOM { get; private set; } + /// <summary> + /// Optional loader arguments object for the plugin + /// </summary> + protected JsonElement? LoaderArgs { get; private set; } + + /// <summary> + /// The path of the plugin's configuration file. (Default = pluginPath.json) + /// </summary> + public string PluginConfigPath { get; init; } + /// <summary> + /// Creates a new <see cref="RuntimePluginLoader"/> with the specified + /// assembly location and host config. + /// </summary> + /// <param name="pluginPath"></param> + /// <param name="log">A nullable log provider</param> + /// <param name="hostConfig">The configuration DOM to merge with plugin config DOM and pass to enabled plugins</param> + /// <param name="unloadable">A value that specifies if the assembly can be unloaded</param> + /// <param name="hotReload">A value that spcifies if the loader will listen for changes to the assembly file and reload the plugins</param> + /// <param name="lazy">A value that specifies if assembly dependencies are loaded on-demand</param> + /// <remarks> + /// The <paramref name="log"/> argument may be null if <paramref name="unloadable"/> is false + /// </remarks> + /// <exception cref="ArgumentNullException"></exception> + public RuntimePluginLoader(string pluginPath, JsonDocument? hostConfig = null, ILogProvider? log = null, bool unloadable = false, bool hotReload = false, bool lazy = false) + :this( + new PluginConfig(pluginPath) + { + IsUnloadable = unloadable || hotReload, + EnableHotReload = hotReload, + IsLazyLoaded = lazy, + ReloadDelay = TimeSpan.FromSeconds(1), + PreferSharedTypes = true, + DefaultContext = AssemblyLoadContext.Default + }, + hostConfig, log) + { + } + /// <summary> + /// Creates a new <see cref="RuntimePluginLoader"/> with the specified config and host config dom. + /// </summary> + /// <param name="config">The plugin's loader configuration </param> + /// <param name="hostConfig">The host/process configuration DOM</param> + /// <param name="log">A log provider to write plugin unload log events to</param> + /// <exception cref="ArgumentNullException"></exception> + public RuntimePluginLoader(PluginConfig config, JsonDocument? hostConfig, ILogProvider? log) + { + //Add the assembly from which the IPlugin library was loaded from + config.SharedAssemblies.Add(typeof(IPlugin).Assembly.GetName()); + + //Default to empty config if null + HostConfig = hostConfig ?? JsonDocument.Parse("{}"); + Loader = new(config); + PluginPath = config.MainAssemblyPath; + Log = log; + Loader.Reloaded += Loader_Reloaded; + //Set the config path default + PluginConfigPath = Path.ChangeExtension(PluginPath, ".json"); + LoadedPlugins = new(); + } + + private async void Loader_Reloaded(object sender, PluginReloadedEventArgs eventArgs) + { + try + { + //Invoke reloaded events + OnBeforeReloaded?.Invoke(this, eventArgs); + //Unload all endpoints + LoadedPlugins.TryForeach(lp => lp.UnloadPlugin(Log)); + //Clear list of loaded plugins + LoadedPlugins.Clear(); + //Unload the plugin config + PluginConfigDOM?.Dispose(); + //Reload the assembly and + await InitLoaderAsync(); + //fire after loaded + OnAfterReloaded?.Invoke(this, eventArgs); + //Raise the external reloaded event + Reloaded?.Invoke(this, EventArgs.Empty); + } + catch (Exception ex) + { + Log?.Error(ex); + } + } + + /// <summary> + /// Initializes the plugin loader, the assembly, and all public <see cref="IPlugin"/> + /// types + /// </summary> + /// <returns>A task that represents the initialization</returns> + public async Task InitLoaderAsync() + { + //Load the main assembly + Assembly PluginAsm = Loader.LoadDefaultAssembly(); + //Get the plugin's configuration file + if (FileOperations.FileExists(PluginConfigPath)) + { + //Open and read the config file + await using FileStream confStream = File.OpenRead(PluginConfigPath); + JsonDocumentOptions jdo = new() + { + AllowTrailingCommas = true, + CommentHandling = JsonCommentHandling.Skip, + }; + //parse the plugin config file + PluginConfigDOM = await JsonDocument.ParseAsync(confStream, jdo); + //Store the config loader args + if (PluginConfigDOM.RootElement.TryGetProperty("loader_args", out JsonElement loaderEl)) + { + LoaderArgs = loaderEl; + } + } + else + { + //Set plugin config dom to an empty object if the file does not exist + PluginConfigDOM = JsonDocument.Parse("{}"); + LoaderArgs = null; + } + + string[] cliArgs = Environment.GetCommandLineArgs(); + + //Get all types that implement the IPlugin interface + IEnumerable<IPlugin> plugins = PluginAsm.GetTypes().Where(static type => !type.IsAbstract && typeof(IPlugin).IsAssignableFrom(type)) + //Create the plugin instances + .Select(static type => (Activator.CreateInstance(type) as IPlugin)!); + //Load all plugins that implement the Iplugin interface + foreach (IPlugin plugin in plugins) + { + //Load wrapper + LivePlugin lp = new(plugin); + try + { + //Init config + lp.InitConfig(HostConfig, PluginConfigDOM); + //Init log handler + lp.InitLog(cliArgs); + //Load the plugin + lp.LoadPlugin(); + //Create new plugin loader for the plugin + LoadedPlugins.AddLast(lp); + } + catch (TargetInvocationException te) when (te.InnerException is not null) + { + throw te.InnerException; + } + } + } + /// <summary> + /// Manually reload the internal <see cref="PluginLoader"/> + /// which will reload the assembly and its plugins and endpoints + /// </summary> + public void ReloadPlugin() => Loader.Reload(); + + /// <summary> + /// Attempts to unload all plugins. + /// </summary> + /// <exception cref="AggregateException"></exception> + public void UnloadAll() => LoadedPlugins.TryForeach(lp => lp.UnloadPlugin(Log)); + + ///<inheritdoc/> + protected override void Free() + { + Loader.Dispose(); + PluginConfigDOM?.Dispose(); + } + + } +}
\ No newline at end of file diff --git a/Plugins.Runtime/src/VNLib.Plugins.Runtime.csproj b/Plugins.Runtime/src/VNLib.Plugins.Runtime.csproj new file mode 100644 index 0000000..d435245 --- /dev/null +++ b/Plugins.Runtime/src/VNLib.Plugins.Runtime.csproj @@ -0,0 +1,47 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <Nullable>enable</Nullable> + <TargetFramework>net6.0</TargetFramework> + <Authors>Vaughn Nugent</Authors> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <Description>A runtime plugin loader for .NET. Allows runtime loading and tracking of .NET assemblies +that export the VNLib.Plugin.IPlugin interface.</Description> + <Version>1.0.1.1</Version> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <!-- Resolve nuget dll files and store them in the output dir --> + <PropertyGroup> + <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <AnalysisLevel>latest-all</AnalysisLevel> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="McMaster.NETCore.Plugins" Version="1.4.0" /> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\..\Essentials\src\VNLib.Plugins.Essentials.csproj" /> + <ProjectReference Include="..\..\Utils\src\VNLib.Utils.csproj" /> + </ItemGroup> + +</Project> diff --git a/Plugins/LICENSE.txt b/Plugins/LICENSE.txt new file mode 100644 index 0000000..2848520 --- /dev/null +++ b/Plugins/LICENSE.txt @@ -0,0 +1,346 @@ +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). + +SPDX-License-Identifier: GPL-2.0-or-later + +License-Text: + +GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + VNLib.Hashing.Portable is a compact .NET managed cryptographic operation + utilities library. + Copyright (C) 2022 Vaughn Nugent + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License.
\ No newline at end of file diff --git a/Plugins/README.md b/Plugins/README.md new file mode 100644 index 0000000..c08107a --- /dev/null +++ b/Plugins/README.md @@ -0,0 +1,81 @@ +# VNLib.Plugins + +A compact and simple contract interface and supporting types for runtime loadable plugins. It implements types that may be used outside the scope of **VNLib.Plugins.Essentials** library, but the contract types were designed for use with it, which is why they are opinionated. + +**This library has no internal or external dependencies** + +### Usage with VNLib.Plugins.Essentials. + +The **VNLib.Plugins.Essentials** library discovers IEndpoint types at runtime inside `EventProcessor` types. For correct usage with the `EventProcessor` class you should implement the interface `IVirtualEndpoint<HttpEntity>` otherwise your endpoint will not get loaded. + +All plugins that intend to be loaded with the **VNLib.Plugins.Essentials.ServiceStack** application library, should conform to these signatures. The ServiceStack library also implements the additional lifecycle hooks if you choose to implement them. + +## Breakdown + +The `IPlugin` interface: a simple contract +``` programming language C# + //Should be public and concrete for runtime loading + public sealed class MyPlugin : IPluign + { + private readonly LinkedList<IEndpoint> _endpoints; + + public MyPlugin() + { + _endpoints = new LinkedList<IEndpoint>(); + } + + public string PluginName { get; } = "MyPlugin"; + + public void Load() + { + //Load plugin, build endpoints + IEndpoint ep1 = new MyEndpoint(); + IEndpoint ep2 = new MyEndpoint(); + //Add endpoints + _endpoints.AddLast(ep1); + _endpoints.AddLast(ep2); + } + + public void Unload() + { + //Unload resources + } + + public IEnumerable<IEndpoint> GetEndpoints() + { + //Return the endpoints for this plugin + return _endpoints; + } + + ... Additional lifecycle methods using the Attributes namespace + } +``` + +The `IEndpoint` interface: represents a resource location in a url search path +``` programming language C# + //Should be public and concrete + internal class MyEndpoint : IEndpoint + { + public string Path { get; } = "/my/resource"; + } +``` + +A step farther is the `IVirtualEndpoint<TEntity>` which processes an entity of the specified type, and implements the IEndpoint interface: +_This interface was built for usage with the `IHttpEvent` interface as the entity type._ +``` + //Should be public and concrete + internal class MyVirtualEndpoint : IVirtualEndpoint<IHttpEvent> + { + public string Path { get; } = "/my/resource"; + + //process HTTP connection + public ValutTask<VfReturnType> Process(IHttpEvent entity) + { + //Process the entity + return VfReturnType.ProcessAsFile; + } + } +``` +## License +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). +See the LICENSE files for more information.
\ No newline at end of file diff --git a/Plugins/src/Attributes/ConfigurationInitalizerAttribute.cs b/Plugins/src/Attributes/ConfigurationInitalizerAttribute.cs new file mode 100644 index 0000000..f214d2b --- /dev/null +++ b/Plugins/src/Attributes/ConfigurationInitalizerAttribute.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: ConfigurationInitalizerAttribute.cs +* +* ConfigurationInitalizerAttribute.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; + +namespace VNLib.Plugins.Attributes +{ + /// <summary> + /// Set this attribute on an <see cref="IPlugin"/> instance method to define the configuration initializer. + /// This attribute can only be defined on a single instance method and cannot be overloaded. + /// <br></br> + /// A plugin host should invoke this method before <see cref="IPlugin.Load"/> + /// <br></br> + /// Method signature <code>public void [methodname] (<see cref="JsonDocument"/> config)</code> + /// </summary> + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)] + public sealed class ConfigurationInitalizerAttribute : Attribute + { } + + /// <summary> + /// Represents a safe configuration initializer delegate method + /// </summary> + /// <param name="config">The configuration object that plugin will use</param> + public delegate void ConfigInitializer(JsonDocument config); +} diff --git a/Plugins/src/Attributes/ConsoleEventHandler.cs b/Plugins/src/Attributes/ConsoleEventHandler.cs new file mode 100644 index 0000000..f3bd061 --- /dev/null +++ b/Plugins/src/Attributes/ConsoleEventHandler.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: ConsoleEventHandler.cs +* +* ConsoleEventHandler.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Plugins.Attributes +{ + /// <summary> + /// <para> + /// Set this attribute on an <see cref="IPlugin"/> instance method to define the console message event handler + /// This attribute can only be defined on a single instance method and cannot be overloaded. + /// </para> + /// <para> + /// Method signature <code>public void [methodname] (<see cref="string"/> command)</code> + /// </para> + /// </summary> + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)] + public class ConsoleEventHandlerAttribute : Attribute + {} + + /// <summary> + /// Represents a safe console event delegate method + /// </summary> + /// <param name="command">The command to be passed to the plugin</param> + public delegate void ConsoleEventHandler(string command); +} diff --git a/Plugins/src/Attributes/LogInitializerAttribute.cs b/Plugins/src/Attributes/LogInitializerAttribute.cs new file mode 100644 index 0000000..61264fb --- /dev/null +++ b/Plugins/src/Attributes/LogInitializerAttribute.cs @@ -0,0 +1,46 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: LogInitializerAttribute.cs +* +* LogInitializerAttribute.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Plugins.Attributes +{ + /// <summary> + /// Set this attribute on an <see cref="IPlugin"/> instance method to define the log initalizer. + /// This attribute can only be defined on a single instance method and cannot be overloaded. + /// <br></br> + /// A plugin host should invoke this method before <see cref="IPlugin.Load"/> but after a <see cref="ConfigurationInitalizerAttribute"/> method + /// <br></br> + /// Method signature <code>public void [methodname] (<see cref="string"/>[] cmdArgs)</code> + /// </summary> + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)] + public sealed class LogInitializerAttribute : Attribute + { } + + /// <summary> + /// Represents a safe logger initializer delegate method + /// </summary> + /// <param name="args">The arguments to pass to the log iniializer (usually command line args)</param> + public delegate void LogInitializer(string[] args); +} diff --git a/Plugins/src/IEndpoint.cs b/Plugins/src/IEndpoint.cs new file mode 100644 index 0000000..33d49df --- /dev/null +++ b/Plugins/src/IEndpoint.cs @@ -0,0 +1,37 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: IEndpoint.cs +* +* IEndpoint.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Plugins +{ + /// <summary> + /// A base class for all entity processing endpoints to listen for requests + /// </summary> + public interface IEndpoint + { + /// <summary> + /// The location path for which to match this handler + /// </summary> + public string Path { get; } + } +}
\ No newline at end of file diff --git a/Plugins/src/IPlugin.cs b/Plugins/src/IPlugin.cs new file mode 100644 index 0000000..16bd403 --- /dev/null +++ b/Plugins/src/IPlugin.cs @@ -0,0 +1,53 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: IPlugin.cs +* +* IPlugin.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; + +namespace VNLib.Plugins +{ + /// <summary> + /// Allows for applications to define plugin capabilities + /// </summary> + public interface IPlugin + { + /// <summary> + /// The name of the plugin to referrence (may be used by the host to interact) + /// </summary> + string PluginName { get; } + /// <summary> + /// Performs operations to prepare the plugin for use + /// </summary> + void Load(); + /// <summary> + /// Invoked when the plugin is unloaded from the runtime + /// </summary> + void Unload(); + /// <summary> + /// Returns all endpoints within the plugin to load into the current root + /// </summary> + /// <returns>An enumeration of endpoints to load</returns> + IEnumerable<IEndpoint> GetEndpoints(); + } +}
\ No newline at end of file diff --git a/Plugins/src/IVirtualEndpoint.cs b/Plugins/src/IVirtualEndpoint.cs new file mode 100644 index 0000000..5f33c0f --- /dev/null +++ b/Plugins/src/IVirtualEndpoint.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: IVirtualEndpoint.cs +* +* IVirtualEndpoint.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +using System.Threading.Tasks; + +namespace VNLib.Plugins +{ + + /// <summary> + /// Represents a virtual page which provides processing on an entity + /// </summary> + /// <typeparam name="TEntity">The entity type to process</typeparam> + public interface IVirtualEndpoint<TEntity> : IEndpoint + { + /// <summary> + /// The handler method for processing the specified location. + /// </summary> + /// <param name="entity">The current connection/session </param> + /// <returns>A <see cref="VfReturnType"/> specifying how the caller should continue processing the request</returns> + public ValueTask<VfReturnType> Process(TEntity entity); + } +}
\ No newline at end of file diff --git a/Plugins/src/VNLib.Plugins.csproj b/Plugins/src/VNLib.Plugins.csproj new file mode 100644 index 0000000..76d292f --- /dev/null +++ b/Plugins/src/VNLib.Plugins.csproj @@ -0,0 +1,35 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <RootNamespace>VNLib.Plugins</RootNamespace> + <Authors>Vaughn Nugent</Authors> + <Company>$(Authors)</Company> + <Product>VNLib Plugins Interface Assembly</Product> + <Description>Provides a standard interface for building dynamically loadable +plugins and asynchronus web endpoint processing, compatible +with the VNLib.Plugins.Runtime loader library.</Description> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <Version>1.0.1.3</Version> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <AssemblyName>VNLib.Plugins</AssemblyName> + <PackageTags>Plugins, VNLIb, VNLib Plugins, Plugin Base</PackageTags> + <Nullable>enable</Nullable> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <AnalysisLevel>latest-all</AnalysisLevel> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> + +</Project> diff --git a/Plugins/src/VfReturnType.cs b/Plugins/src/VfReturnType.cs new file mode 100644 index 0000000..8ebcb26 --- /dev/null +++ b/Plugins/src/VfReturnType.cs @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: Interfaces.cs +* +* Interfaces.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Plugins +{ + + /// <summary> + /// Represents the result of a virutal endpoint processing operation + /// </summary> + public enum VfReturnType + { + /// <summary> + /// Signals that the virtual endpoint + /// </summary> + ProcessAsFile, + /// <summary> + /// Signals that the virtual endpoint generated a response, and + /// the connection should be completed + /// </summary> + VirtualSkip, + /// <summary> + /// Signals that the virtual endpoint determined that the connection + /// should be denied. + /// </summary> + Forbidden, + /// <summary> + /// Signals that the resource the virtual endpoint was processing + /// does not exist. + /// </summary> + NotFound, + /// <summary> + /// Signals that the virutal endpoint determined the request was invalid + /// </summary> + BadRequest, + /// <summary> + /// Signals that the virtual endpoint had an error + /// </summary> + Error + } +} diff --git a/Plugins/src/WebMessage.cs b/Plugins/src/WebMessage.cs new file mode 100644 index 0000000..fb6ca6f --- /dev/null +++ b/Plugins/src/WebMessage.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Plugins +* File: WebMessage.cs +* +* WebMessage.cs is part of VNLib.Plugins which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Plugins is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Plugins is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Plugins. If not, see http://www.gnu.org/licenses/. +*/ + +using System.Text.Json.Serialization; + +namespace VNLib.Plugins +{ + public class WebMessage + { + /// <summary> + /// The encrypted access token for the client to use after a login request + /// </summary> + [JsonPropertyName("token")] + public string? Token { get; set; } + /// <summary> + /// The result of the REST operation to send to client + /// </summary> + [JsonPropertyName("result")] + public object? Result { get; set; } + /// <summary> + /// A status flag/result of the REST operation + /// </summary> + [JsonPropertyName("success")] + public bool Success { get; set; } + } +} diff --git a/README.md b/README.md new file mode 100644 index 0000000..7367c15 --- /dev/null +++ b/README.md @@ -0,0 +1,30 @@ +# VNLib.Core +<hr/> + +*A Mono-repo for "core" VNLib projects for simplicity.* + +### What is VNLib.Core? <hr/> + +VNLib is a collection of .NET/C# (mostly) libraries that I maintain for many private and public projects. VNLib.Core is a subset of the larger collection of projects I have named VNLib. This repo contains libraries I consider to be utility only, or building blocks that are individually useful for other projects. You will likely see many or most of them used across may other VNLib type projects. These libraries are meant to be stand-alone, meaning that there are no required* external dependencies (except the **[mscorelib](https://github.com/dotnet/runtime)**). For example the VNLib.Utils library is a standalone, 0 required dependency library that is useful for, logging, common extensions, and a significant collection of memory related utilities. + +Any libraries in this repository that contain external dependencies will be mentioned explicitly in the library's readme. I intend to limit this behavior, as it is the reason this repository exists. + +### How is this repository maintained? <hr/> +I use many internal tools to build and maintain these projects. I use [OneDev](https://code.onedev.io/) for my internal source control and updates are pushed to GitHub as part of a build process. I use [Task](https://taskfile.dev) to maintain the build/release for all projects in this repository. Builds are publicly available on my [website](https://www.vaughnnugent.com/resources/software) described in the builds section. I do *not* intend to expose my internal tools for security reasons. + +### Licensing <hr/> +Projects contained in this repository are individually licensed, either GNU GPL V2+ or GNU GPL GPL Aferro V3+. Builds contain the required license txt files in the archive. + +### Why so many individual projects? <hr/> +Motivation: I work on many different types of projects and often require modules that I have built before but haven't maintained etc. To solve this issue I prioritize code-reuse by maintaining a smaller subset of libraries that I may include as-needed (similar to how .NET core/5+ maintains libraries outside of mscorelib). I also have a strong dislike for including projects that require many external dependencies for a simple feature/function. I would prefer the ability to include the least amount if code required to get the job done with the least amount of side-effects. Finally I prefer homogeneity in my projects so I tend to use .NET implementations opposed to 3rd party even if the implementation is not as performant for my use-cases. + +### Builds <hr/> +Builds contain the individual components listed below packaged per-project, available for download on my [website](https://www.vaughnnugent.com/resources/software). Builds are maintained manually for now until I build a testing pipeline. Build packages will be tar +gzipped (except for nuget packages). + +*All downloads will contain a sha384 hash of the file by adding a .sha384 to the desired file download, eg: debug.tgz.sha384* +*PGP signed downloads will be available eventually* + +- Project source code (src.tgz) +- Nuget package (where applicable), debug w/ symbols & source + release (pkg/buildType/projName.version.nupkg) +- Debug build w/ symbols & xml docs (debug.tgz) +- Release build (release.tgz)
\ No newline at end of file diff --git a/Utils/LICENSE.txt b/Utils/LICENSE.txt new file mode 100644 index 0000000..cbb3969 --- /dev/null +++ b/Utils/LICENSE.txt @@ -0,0 +1,293 @@ +Copyright (c) 2022 Vaughn Nugent + +Contact information + Name: Vaughn Nugent + Email: public[at]vaughnnugent[dot]com + Website: https://www.vaughnnugent.com + +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). + +SPDX-License-Identifier: GPL-2.0-or-later + +License-Text: + +GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS
\ No newline at end of file diff --git a/Utils/README.md b/Utils/README.md new file mode 100644 index 0000000..ddd7e84 --- /dev/null +++ b/Utils/README.md @@ -0,0 +1,36 @@ +# VNLib.Utils + +A .NET/C# library for common .NET operation optimizations. + +namespaces +- VNLib.Utils.Async - Provides classes for asynchronous access synchronization and some asynchronous collections +- VNLib.Utils.Extensions - Internal and external extensions for condensed common operations +- VNLib.Utils.IO - Memory focused data structures for IO operations +- VNLib.Utils.Logging - Logging interfaces for zero dependency logging +- VNLib.Utils.Memory - Utilities for safely accessing unmanaged memory and CLR memory +- VNLib.Utils.Memory.Caching - Data structures for managed object, data caching, and interfaces for cache-able objects + + +### Recommended 3rd Party Libs + +This library does not *require* any direct dependencies, however there are some optional ones that are desired for higher performance code. This library does not, modify, contribute, or affect the functionality of any of the 3rd party libraries recommended below. + +[**RPMalloc**](https://github.com/mjansson/rpmalloc) By Mattias Jansson - VNlib.Utils.Memory (and sub-classes) may load and bind function calls to this native library determined by environment variables. To use RPMalloc as the default unmanaged allocator simply add the dynamic library to the native lib search path, such as in the executable directory, and set the allocator environment variable as instructed below. + + +### Allocator selection via environment variables +Valid allocator value for the `VNLIB_SHARED_HEAP_TYPE` environment variable: +- "win32" - for win32 based private heaps (only valid if using the Microsoft Windows operating system) +- "rpmalloc" - to load the RPMalloc native library if compiled for your platform +- none - the default value, will attempt to load the win32 private heap Kernel32 library, otherwise, the native ProcessHeap() cross platform allocator + + +## Usage +A usage breakdown would be far to lengthy for this library, and instead I intend to keep valid and comprehensive documentation in Visual Studio XML documentation files included in this project's src directory. + +This library is a utilities library and therefor may be directly included in your application or libraries, + +### License + +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). +See the LICENSE files for more information.
\ No newline at end of file diff --git a/Utils/src/Async/AccessSerializer.cs b/Utils/src/Async/AccessSerializer.cs new file mode 100644 index 0000000..ce78f6c --- /dev/null +++ b/Utils/src/Async/AccessSerializer.cs @@ -0,0 +1,297 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: AccessSerializer.cs +* +* AccessSerializer.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// Provides access arbitration to an exclusive resouce + /// </summary> + /// <typeparam name="TKey">The uinique identifier type for the resource</typeparam> + /// <typeparam name="TResource">The resource type</typeparam> + public sealed class AccessSerializer<TKey, TResource> where TResource : IExclusiveResource + { + private readonly SemaphoreSlim semaphore; + private readonly Func<TKey, TResource> Factory; + private readonly Action CompletedCb; + private int WaitingCount; + /// <summary> + /// Creates a new <see cref="AccessSerializer{K, T}"/> with the specified factory and completed callback + /// </summary> + /// <param name="factory">Factory function to genereate new <typeparamref name="TResource"/> objects from <typeparamref name="TKey"/> keys</param> + /// <param name="completedCb">Function to be invoked when the encapsulated objected is no longer in use</param> + /// <exception cref="ArgumentNullException"></exception> + public AccessSerializer(Func<TKey, TResource> factory, Action completedCb) + { + this.Factory = factory ?? throw new ArgumentNullException(nameof(factory)); + this.CompletedCb = completedCb; + //Setup semaphore for locking + this.semaphore = new SemaphoreSlim(1, 1); + this.WaitingCount = 0; + } + + /// <summary> + /// Attempts to obtain an exclusive lock on the object + /// </summary> + /// <param name="key"></param> + /// <param name="wait">Time to wait for lock</param> + /// <param name="exObj"></param> + /// <returns>true if lock was obtained within the timeout, false if the lock was not obtained</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public bool TryWait(TKey key, TimeSpan wait, out ExclusiveResourceHandle<TResource> exObj) + { + //Increase waiting count while we wait + Interlocked.Increment(ref WaitingCount); + try + { + //Try to obtain the lock + if (semaphore.Wait(wait)) + { + TResource get() => Factory(key); + //Create new exclusive lock handle that will generate a new that calls release when freed + exObj = new(get, Release); + return true; + } + //Lock not taken + exObj = null; + return false; + } + finally + { + //Decrease the waiting count since we are no longer waiting + Interlocked.Decrement(ref WaitingCount); + } + } + /// <summary> + /// Waits for exclusive access to the resource. + /// </summary> + /// <param name="key"></param> + /// <returns>An <see cref="ExclusiveResourceHandle{T}"/> encapsulating the resource</returns> + public ExclusiveResourceHandle<TResource> Wait(TKey key) + { + try + { + //Increase waiting count while we wait + Interlocked.Increment(ref WaitingCount); + //Try to obtain the lock + semaphore.Wait(); + //Local function to generate the output value + TResource get() => Factory(key); + //Create new exclusive lock handle that will generate a new that calls release when freed + return new(get, Release); + } + finally + { + //Decrease the waiting count since we are no longer waiting + Interlocked.Decrement(ref WaitingCount); + } + } + /// <summary> + /// Asynchronously waits for exclusive access to the resource. + /// </summary> + /// <returns>An <see cref="ExclusiveResourceHandle{TResource}"/> encapsulating the resource</returns> + public async Task<ExclusiveResourceHandle<TResource>> WaitAsync(TKey key, CancellationToken cancellationToken = default) + { + try + { + //Increase waiting count while we wait + Interlocked.Increment(ref WaitingCount); + //Try to obtain the lock + await semaphore.WaitAsync(cancellationToken); + //Local function to generate the output value + TResource get() => Factory(key); + //Create new exclusive lock handle that will generate a new that calls release when freed + return new(get, Release); + } + finally + { + //Decrease the waiting count since we are no longer waiting + Interlocked.Decrement(ref WaitingCount); + } + } + /// <summary> + /// Releases an exclusive lock that is held on an object + /// </summary> + private void Release() + { + /* + * If objects are waiting for the current instance, then we will release + * the semaphore and exit, as we no longer have control over the context + */ + if (WaitingCount > 0) + { + this.semaphore.Release(); + } + else + { + //Do not release the sempahore, just dispose of the semaphore + this.semaphore.Dispose(); + //call the completed function + CompletedCb?.Invoke(); + } + } + } + + /// <summary> + /// Provides access arbitration to an <see cref="IExclusiveResource"/> + /// </summary> + /// <typeparam name="TKey">The uinique identifier type for the resource</typeparam> + /// <typeparam name="TArg">The type of the optional argument to be passed to the user-implemented factory function</typeparam> + /// <typeparam name="TResource">The resource type</typeparam> + public sealed class AccessSerializer<TKey, TArg, TResource> where TResource : IExclusiveResource + { + private readonly SemaphoreSlim semaphore; + private readonly Func<TKey, TArg, TResource> Factory; + private readonly Action CompletedCb; + private int WaitingCount; + /// <summary> + /// Creates a new <see cref="AccessSerializer{TKey, TArg, TResource}"/> with the specified factory and completed callback + /// </summary> + /// <param name="factory">Factory function to genereate new <typeparamref name="TResource"/> objects from <typeparamref name="TKey"/> keys</param> + /// <param name="completedCb">Function to be invoked when the encapsulated objected is no longer in use</param> + /// <exception cref="ArgumentNullException"></exception> + public AccessSerializer(Func<TKey, TArg, TResource> factory, Action completedCb) + { + this.Factory = factory ?? throw new ArgumentNullException(nameof(factory)); + this.CompletedCb = completedCb; + //Setup semaphore for locking + this.semaphore = new SemaphoreSlim(1, 1); + this.WaitingCount = 0; + } + + /// <summary> + /// Attempts to obtain an exclusive lock on the object + /// </summary> + /// <param name="key"></param> + /// <param name="arg">The key identifying the resource</param> + /// <param name="wait">Time to wait for lock</param> + /// <param name="exObj"></param> + /// <returns>true if lock was obtained within the timeout, false if the lock was not obtained</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public bool TryWait(TKey key, TArg arg, TimeSpan wait, out ExclusiveResourceHandle<TResource> exObj) + { + //Increase waiting count while we wait + Interlocked.Increment(ref WaitingCount); + try + { + //Try to obtain the lock + if (semaphore.Wait(wait)) + { + TResource get() => Factory(key, arg); + //Create new exclusive lock handle that will generate a new that calls release when freed + exObj = new(get, Release); + return true; + } + //Lock not taken + exObj = null; + return false; + } + finally + { + //Decrease the waiting count since we are no longer waiting + Interlocked.Decrement(ref WaitingCount); + } + } + /// <summary> + /// Waits for exclusive access to the resource. + /// </summary> + /// <param name="key">The unique key that identifies the resource</param> + /// <param name="arg">The state argument to pass to the factory function</param> + /// <returns>An <see cref="ExclusiveResourceHandle{TResource}"/> encapsulating the resource</returns> + public ExclusiveResourceHandle<TResource> Wait(TKey key, TArg arg) + { + try + { + //Increase waiting count while we wait + Interlocked.Increment(ref WaitingCount); + //Try to obtain the lock + semaphore.Wait(); + //Local function to generate the output value + TResource get() => Factory(key, arg); + //Create new exclusive lock handle that will generate a new that calls release when freed + return new(get, Release); + } + finally + { + //Decrease the waiting count since we are no longer waiting + Interlocked.Decrement(ref WaitingCount); + } + } + /// <summary> + /// Asynchronously waits for exclusive access to the resource. + /// </summary> + /// <param name="key"></param> + /// <param name="arg">The state argument to pass to the factory function</param> + /// <param name="cancellationToken"></param> + /// <returns>An <see cref="ExclusiveResourceHandle{TResource}"/> encapsulating the resource</returns> + public async Task<ExclusiveResourceHandle<TResource>> WaitAsync(TKey key, TArg arg, CancellationToken cancellationToken = default) + { + try + { + //Increase waiting count while we wait + Interlocked.Increment(ref WaitingCount); + //Try to obtain the lock + await semaphore.WaitAsync(cancellationToken); + //Local function to generate the output value + TResource get() => Factory(key, arg); + //Create new exclusive lock handle that will generate a new that calls release when freed + return new(get, Release); + } + finally + { + //Decrease the waiting count since we are no longer waiting + Interlocked.Decrement(ref WaitingCount); + } + } + + /// <summary> + /// Releases an exclusive lock that is held on an object + /// </summary> + private void Release() + { + /* + * If objects are waiting for the current instance, then we will release + * the semaphore and exit, as we no longer have control over the context + */ + if (WaitingCount > 0) + { + this.semaphore.Release(); + } + else + { + //Do not release the sempahore, just dispose of the semaphore + this.semaphore.Dispose(); + //call the completed function + CompletedCb?.Invoke(); + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/Async/AsyncExclusiveResource.cs b/Utils/src/Async/AsyncExclusiveResource.cs new file mode 100644 index 0000000..18e2a42 --- /dev/null +++ b/Utils/src/Async/AsyncExclusiveResource.cs @@ -0,0 +1,169 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: AsyncExclusiveResource.cs +* +* AsyncExclusiveResource.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// Provides a base class for resources that must be obtained exclusivly in a multi-threaded environment + /// but allow state update operations (and their exceptions) to be deferred to the next accessor. + /// </summary> + /// <typeparam name="TState">The state parameter type passed during updates</typeparam> + public abstract class AsyncExclusiveResource<TState> : VnDisposeable, IWaitHandle, IAsyncWaitHandle + { + /// <summary> + /// Main mutli-threading lock used for primary access synchronization + /// </summary> + protected SemaphoreSlim MainLock { get; } = new (1, 1); + + private Task? LastUpdate; + + /// <summary> + /// <inheritdoc/> + /// <br></br> + /// <br></br> + /// If the previous call to <see cref="UpdateAndRelease"/> resulted in an asynchronous update, and exceptions occured, an <see cref="AsyncUpdateException"/> + /// will be thrown enclosing the exception + /// </summary> + /// <param name="millisecondsTimeout">Time in milliseconds to wait for exclusive access to the resource</param> + /// <exception cref="AsyncUpdateException"></exception> + /// <inheritdoc/> + public virtual bool WaitOne(int millisecondsTimeout) + { + //First wait for main lock + if (MainLock.Wait(millisecondsTimeout)) + { + //Main lock has been taken + try + { + //Wait for async update if there is one pending(will throw exceptions if any occurred) + LastUpdate?.Wait(); + return true; + } + catch (AggregateException ae) when (ae.InnerException != null) + { + //Release the main lock and re-throw the inner exception + _ = MainLock.Release(); + //Throw a new async update exception + throw new AsyncUpdateException(ae.InnerException); + } + catch + { + //Release the main lock and re-throw the exception + _ = MainLock.Release(); + throw; + } + } + return false; + } + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public virtual async Task WaitOneAsync(CancellationToken token = default) + { + //Wait for main lock + await MainLock.WaitAsync(token).ConfigureAwait(true); + //if the last update completed synchronously, return true + if (LastUpdate == null) + { + return; + } + try + { + //Await the last update task and catch its exceptions + await LastUpdate.ConfigureAwait(false); + } + catch + { + //Release the main lock and re-throw the exception + _ = MainLock.Release(); + throw; + } + } + + /// <summary> + /// Requests a resource update and releases the exclusive lock on this resource. If a deferred update operation has any + /// exceptions during its last operation, they will be thrown here. + /// </summary> + /// <param name="defer">Specifies weather the update should be deferred or awaited on the current call</param> + /// <param name="state">A state parameter to be pased to the update function</param> + /// <exception cref="ObjectDisposedException"></exception> + public async ValueTask UpdateAndRelease(bool defer, TState state) + { + //Otherwise wait and update on the current thread + try + { + //Dispose the update task + LastUpdate?.Dispose(); + //Remove the referrence + LastUpdate = null; + //Run update on the current thread + LastUpdate = await UpdateResource(defer, state).ConfigureAwait(true); + //If the update is not deferred, await the results + if (!defer && LastUpdate != null) + { + await LastUpdate.ConfigureAwait(true); + } + } + finally + { + //Release the main lock + _ = MainLock.Release(); + } + } + + /// <summary> + /// <para> + /// When overrriden in a derived class, is responsible for updating the state of the instance if necessary. + /// </para> + /// <para> + /// If the result of the update retruns a <see cref="Task"/> that represents the deferred update, the next call to <see cref="WaitOne"/> will + /// block until the operation completes and will throw any exceptions that occured + /// </para> + /// </summary> + /// <param name="defer">true if the caller expects a resource update to be deferred, false if the caller expects the result of the update to be awaited</param> + /// <param name="state">State parameter passed when releasing</param> + /// <returns>A <see cref="Task"/> representing the async state update operation, or null if no async state update operation need's to be monitored</returns> + protected abstract ValueTask<Task?> UpdateResource(bool defer, TState state); + + ///<inheritdoc/> + protected override void Free() + { + //Dispose lock + MainLock.Dispose(); + + //Try to cleanup the last update + if (LastUpdate != null && LastUpdate.IsCompletedSuccessfully) + { + LastUpdate.Dispose(); + } + + LastUpdate = null; + } + + } +}
\ No newline at end of file diff --git a/Utils/src/Async/AsyncQueue.cs b/Utils/src/Async/AsyncQueue.cs new file mode 100644 index 0000000..ba45513 --- /dev/null +++ b/Utils/src/Async/AsyncQueue.cs @@ -0,0 +1,144 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: AsyncQueue.cs +* +* AsyncQueue.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using System.Threading.Channels; +using System.Diagnostics.CodeAnalysis; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// Provides a <see cref="Channel{T}"/> based asynchronous queue + /// </summary> + /// <typeparam name="T">The event object type</typeparam> + public class AsyncQueue<T> + { + private readonly Channel<T> _channel; + + /// <summary> + /// Initalizes a new multi-threaded bound channel queue, that accepts + /// the <paramref name="capacity"/> number of items before it will + /// return asynchronously, or fail to enqueue items + /// </summary> + /// <param name="capacity">The maxium number of items to allow in the queue</param> + public AsyncQueue(int capacity):this(false, false, capacity) + {} + /// <summary> + /// Initalizes a new multi-threaded unbound channel queue + /// </summary> + public AsyncQueue():this(false, false) + {} + + /// <summary> + /// Initalizes a new queue that allows specifying concurrency requirements + /// and a bound/unbound channel capacity + /// </summary> + /// <param name="singleWriter">A value that specifies only a single thread be enqueing items?</param> + /// <param name="singleReader">A value that specifies only a single thread will be dequeing</param> + /// <param name="capacity">The maxium number of items to enque without failing</param> + public AsyncQueue(bool singleWriter, bool singleReader, int capacity = int.MaxValue) + { + if(capacity == int.MaxValue) + { + //Create unbounded + UnboundedChannelOptions opt = new() + { + SingleReader = singleReader, + SingleWriter = singleWriter, + AllowSynchronousContinuations = true, + }; + _channel = Channel.CreateUnbounded<T>(opt); + } + else + { + //Create bounded + BoundedChannelOptions opt = new(capacity) + { + SingleReader = singleReader, + SingleWriter = singleWriter, + AllowSynchronousContinuations = true, + //Default wait for space + FullMode = BoundedChannelFullMode.Wait + }; + _channel = Channel.CreateBounded<T>(opt); + } + } + + /// <summary> + /// Initalizes a new unbound channel based queue + /// </summary> + /// <param name="ubOptions">Channel options</param> + public AsyncQueue(UnboundedChannelOptions ubOptions) + { + _channel = Channel.CreateUnbounded<T>(ubOptions); + } + + /// <summary> + /// Initalizes a new bound channel based queue + /// </summary> + /// <param name="options">Channel options</param> + public AsyncQueue(BoundedChannelOptions options) + { + _channel = Channel.CreateBounded<T>(options); + } + + /// <summary> + /// Attemts to enqeue an item if the queue has the capacity + /// </summary> + /// <param name="item">The item to eqneue</param> + /// <returns>True if the queue can accept another item, false otherwise</returns> + public bool TryEnque(T item) => _channel.Writer.TryWrite(item); + /// <summary> + /// Enqueues an item to the end of the queue and notifies a waiter that an item was enqueued + /// </summary> + /// <param name="item">The item to enqueue</param> + /// <param name="cancellationToken"></param> + /// <exception cref="ObjectDisposedException"></exception> + public ValueTask EnqueueAsync(T item, CancellationToken cancellationToken = default) => _channel.Writer.WriteAsync(item, cancellationToken); + /// <summary> + /// Asynchronously waits for an item to be Enqueued to the end of the queue. + /// </summary> + /// <returns>The item at the begining of the queue</returns> + /// <exception cref="ObjectDisposedException"></exception> + public ValueTask<T> DequeueAsync(CancellationToken cancellationToken = default) => _channel.Reader.ReadAsync(cancellationToken); + /// <summary> + /// Removes the object at the beginning of the queue and stores it to the result parameter. Without waiting for a change + /// event. + /// </summary> + /// <param name="result">The item that was at the begining of the queue</param> + /// <returns>True if the queue could be read synchronously, false if the lock could not be entered, or the queue contains no items</returns> + /// <exception cref="ObjectDisposedException"></exception> + public bool TryDequeue([MaybeNullWhen(false)] out T result) => _channel.Reader.TryRead(out result); + /// <summary> + /// Peeks the object at the beginning of the queue and stores it to the result parameter. Without waiting for a change + /// event. + /// </summary> + /// <param name="result">The item that was at the begining of the queue</param> + /// <returns>True if the queue could be read synchronously, false if the lock could not be entered, or the queue contains no items</returns> + /// <exception cref="ObjectDisposedException"></exception> + public bool TryPeek([MaybeNullWhen(false)] out T result) => _channel.Reader.TryPeek(out result); + } +} diff --git a/Utils/src/Async/AsyncUpdatableResource.cs b/Utils/src/Async/AsyncUpdatableResource.cs new file mode 100644 index 0000000..b4ce519 --- /dev/null +++ b/Utils/src/Async/AsyncUpdatableResource.cs @@ -0,0 +1,111 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: AsyncUpdatableResource.cs +* +* AsyncUpdatableResource.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text.Json; +using System.Threading.Tasks; + +using VNLib.Utils.IO; +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// A callback delegate used for updating a <see cref="AsyncUpdatableResource"/> + /// </summary> + /// <param name="source">The <see cref="AsyncUpdatableResource"/> to be updated</param> + /// <param name="data">The serialized data to be stored/updated</param> + /// <exception cref="ResourceUpdateFailedException"></exception> + public delegate Task AsyncUpdateCallback(object source, Stream data); + /// <summary> + /// A callback delegate invoked when a <see cref="AsyncUpdatableResource"/> delete is requested + /// </summary> + /// <param name="source">The <see cref="AsyncUpdatableResource"/> to be deleted</param> + /// <exception cref="ResourceDeleteFailedException"></exception> + public delegate Task AsyncDeleteCallback(object source); + + /// <summary> + /// Implemented by a resource that is backed by an external data store, that when modified or deleted will + /// be reflected to the backing store. + /// </summary> + public abstract class AsyncUpdatableResource : BackedResourceBase, IAsyncExclusiveResource + { + protected abstract AsyncUpdateCallback UpdateCb { get; } + protected abstract AsyncDeleteCallback DeleteCb { get; } + + /// <summary> + /// Releases the resource and flushes pending changes to its backing store. + /// </summary> + /// <returns>A task that represents the async operation</returns> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ResourceDeleteFailedException"></exception> + /// <exception cref="ResourceUpdateFailedException"></exception> + public virtual async ValueTask ReleaseAsync() + { + //If resource has already been realeased, return + if (IsReleased) + { + return; + } + //If deleted flag is set, invoke the delete callback + if (Deleted) + { + await DeleteCb(this).ConfigureAwait(true); + } + //If the state has been modifed, flush changes to the store + else if (Modified) + { + await FlushPendingChangesAsync().ConfigureAwait(true); + } + //Set the released value + IsReleased = true; + } + + /// <summary> + /// <para> + /// Writes the current state of the the resource to the backing store + /// immediatly by invoking the specified callback. + /// </para> + /// <para> + /// Only call this method if your store supports multiple state updates + /// </para> + /// </summary> + protected virtual async Task FlushPendingChangesAsync() + { + //Get the resource + object resource = GetResource(); + //Open a memory stream to store data in + using VnMemoryStream data = new(); + //Serialize and write to stream + VnEncoding.JSONSerializeToBinary(resource, data, resource.GetType(), base.JSO); + //Reset stream to begining + _ = data.Seek(0, SeekOrigin.Begin); + //Invoke update callback + await UpdateCb(this, data).ConfigureAwait(true); + //Clear modified flag + Modified = false; + } + } +} diff --git a/Utils/src/Async/Exceptions/AsyncUpdateException.cs b/Utils/src/Async/Exceptions/AsyncUpdateException.cs new file mode 100644 index 0000000..de5a491 --- /dev/null +++ b/Utils/src/Async/Exceptions/AsyncUpdateException.cs @@ -0,0 +1,52 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: AsyncUpdateException.cs +* +* AsyncUpdateException.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// Represents an exception that was raised during an asyncronous update of a resource. The <see cref="Exception.InnerException"/> stores the + /// details of the actual exception raised + /// </summary> + public sealed class AsyncUpdateException : ResourceUpdateFailedException + { + /// <summary> + /// + /// </summary> + /// <param name="inner"></param> + public AsyncUpdateException(Exception inner) : base("", inner) { } + + public AsyncUpdateException() + {} + + public AsyncUpdateException(string message) : base(message) + {} + + public AsyncUpdateException(string message, Exception innerException) : base(message, innerException) + {} + } +}
\ No newline at end of file diff --git a/Utils/src/Async/IAsyncExclusiveResource.cs b/Utils/src/Async/IAsyncExclusiveResource.cs new file mode 100644 index 0000000..93157ce --- /dev/null +++ b/Utils/src/Async/IAsyncExclusiveResource.cs @@ -0,0 +1,40 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IAsyncExclusiveResource.cs +* +* IAsyncExclusiveResource.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System.Threading.Tasks; +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// <inheritdoc/> + /// </summary> + public interface IAsyncExclusiveResource : IResource + { + /// <summary> + /// Releases the resource from use. Called when a <see cref="ExclusiveResourceHandle{T}"/> is disposed + /// </summary> + ValueTask ReleaseAsync(); + } +}
\ No newline at end of file diff --git a/Utils/src/Async/IAsyncWaitHandle.cs b/Utils/src/Async/IAsyncWaitHandle.cs new file mode 100644 index 0000000..1cadc06 --- /dev/null +++ b/Utils/src/Async/IAsyncWaitHandle.cs @@ -0,0 +1,41 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IAsyncWaitHandle.cs +* +* IAsyncWaitHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// Provides a synchronization handle that can be asynchronously aquired + /// </summary> + public interface IAsyncWaitHandle + { + /// <summary> + /// Waits for exclusive access to the resource until the <see cref="CancellationToken"/> expires + /// </summary> + /// <inheritdoc/> + Task WaitOneAsync(CancellationToken token = default); + } +}
\ No newline at end of file diff --git a/Utils/src/Async/IWaitHandle.cs b/Utils/src/Async/IWaitHandle.cs new file mode 100644 index 0000000..85e8a2a --- /dev/null +++ b/Utils/src/Async/IWaitHandle.cs @@ -0,0 +1,59 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IWaitHandle.cs +* +* IWaitHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; + +namespace VNLib.Utils.Async +{ + /// <summary> + /// Provides basic thread synchronization functions similar to <see cref="WaitHandle"/> + /// </summary> + public interface IWaitHandle + { + /// <summary> + /// Waits for exclusive access to the resource indefinitly. If the signal is never received this method never returns + /// </summary> + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + /// <returns>true if the current thread received the signal</returns> + public virtual bool WaitOne() => WaitOne(Timeout.Infinite); + /// <summary> + /// Waits for exclusive access to the resource until the specified number of milliseconds + /// </summary> + /// <param name="millisecondsTimeout">Time in milliseconds to wait for exclusive access to the resource</param> + /// <returns>true if the current thread received the signal, false if the timout expired, and access was not granted</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + bool WaitOne(int millisecondsTimeout); + /// <summary> + /// Waits for exclusive access to the resource until the specified <see cref="TimeSpan"/> + /// </summary> + /// <returns>true if the current thread received the signal, false if the timout expired, and access was not granted</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public virtual bool WaitOne(TimeSpan timeout) => WaitOne(Convert.ToInt32(timeout.TotalMilliseconds)); + } +}
\ No newline at end of file diff --git a/Utils/src/BitField.cs b/Utils/src/BitField.cs new file mode 100644 index 0000000..bc001df --- /dev/null +++ b/Utils/src/BitField.cs @@ -0,0 +1,115 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: BitField.cs +* +* BitField.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.CompilerServices; + +namespace VNLib.Utils +{ + /// <summary> + /// Represents a field of 64 bits that can be set or cleared using unsigned or signed masks + /// </summary> + public class BitField + { + private ulong Field; + /// <summary> + /// The readonly value of the <see cref="BitField"/> + /// </summary> + public ulong Value => Field; + /// <summary> + /// Creates a new <see cref="BitField"/> initialized to the specified value + /// </summary> + /// <param name="initial">Initial value</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public BitField(ulong initial) => Field = initial; + /// <summary> + /// Creates a new <see cref="BitField"/> initialized to the specified value + /// </summary> + /// <param name="initial">Initial value</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public BitField(long initial) => Field = unchecked((ulong)initial); + /// <summary> + /// Determines if the specified flag is set + /// </summary> + /// <param name="mask">The mask to compare against the field value</param> + /// <returns>True if the flag(s) is currently set, false if flag is not set</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public bool IsSet(ulong mask) => (Field & mask) != 0; + /// <summary> + /// Determines if the specified flag is set + /// </summary> + /// <param name="mask">The mask to compare against the field value</param> + /// <returns>True if the flag(s) is currently set, false if flag is not set</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public bool IsSet(long mask) => (Field & unchecked((ulong)mask)) != 0; + /// <summary> + /// Determines if the specified flag is set + /// </summary> + /// <param name="mask">The mask to compare against the field value</param> + /// <returns>True if the flag(s) is currently set, false if flag is not set</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Set(ulong mask) => Field |= mask; + /// <summary> + /// Determines if the specified flag is set + /// </summary> + /// <param name="mask">The mask to compare against the field value</param> + /// <returns>True if the flag(s) is currently set, false if flag is not set</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Set(long mask) => Field |= unchecked((ulong)mask); + /// <summary> + /// Sets or clears a flag(s) indentified by a mask based on the value + /// </summary> + /// <param name="mask">Mask used to identify flags</param> + /// <param name="value">True to set a flag, false to clear a flag</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Set(ulong mask, bool value) + { + if (value) + { + Set(mask); + } + else + { + Clear(mask); + } + } + /// <summary> + /// Clears the flag identified by the specified mask + /// </summary> + /// <param name="mask">The mask used to clear the given flag</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Clear(ulong mask) => Field &= ~mask; + /// <summary> + /// Clears the flag identified by the specified mask + /// </summary> + /// <param name="mask">The mask used to clear the given flag</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void Clear(long mask) => Field &= ~unchecked((ulong)mask); + /// <summary> + /// Clears all flags by setting the <see cref="Field"/> property value to 0 + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void ClearAll() => Field = 0; + } +}
\ No newline at end of file diff --git a/Utils/src/ERRNO.cs b/Utils/src/ERRNO.cs new file mode 100644 index 0000000..c3c61de --- /dev/null +++ b/Utils/src/ERRNO.cs @@ -0,0 +1,152 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ERRNO.cs +* +* ERRNO.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.InteropServices; + +namespace VNLib.Utils +{ + /// <summary> + /// Implements a C style integer error code type. Size is platform dependent + /// </summary> + [StructLayout(LayoutKind.Sequential)] + public readonly struct ERRNO : IEquatable<ERRNO>, ISpanFormattable, IFormattable + { + /// <summary> + /// Represents a successfull error code (true) + /// </summary> + public static readonly ERRNO SUCCESS = true; + + /// <summary> + /// Represents a failure error code (false) + /// </summary> + public static readonly ERRNO E_FAIL = false; + + private readonly nint ErrorCode; + /// <summary> + /// Creates a new <see cref="ERRNO"/> from the specified error value + /// </summary> + /// <param name="errno">The value of the error to represent</param> + public ERRNO(nint errno) => ErrorCode = errno; + /// <summary> + /// Creates a new <see cref="ERRNO"/> from an <see cref="int"/> error code. null = 0 = false + /// </summary> + /// <param name="errorVal">Error code</param> + public static implicit operator ERRNO(int errorVal) => new (errorVal); + /// <summary> + /// Creates a new <see cref="ERRNO"/> from an <see cref="int"/> error code. null = 0 = false + /// </summary> + /// <param name="errorVal">Error code</param> + public static explicit operator ERRNO(int? errorVal) => new(errorVal ?? 0); + /// <summary> + /// Creates a new <see cref="ERRNO"/> from a booleam, 1 if true, 0 if false + /// </summary> + /// <param name="errorVal"></param> + public static implicit operator ERRNO(bool errorVal) => new(errorVal ? 1 : 0); + /// <summary> + /// Creates a new <see cref="ERRNO"/> from a pointer value + /// </summary> + /// <param name="errno">The pointer value representing an error code</param> + public static implicit operator ERRNO(nint errno) => new(errno); + /// <summary> + /// Error value as integer. Value of supplied error code or if cast from boolean 1 if true, 0 if false + /// </summary> + /// <param name="errorVal"><see cref="ERRNO"/> to get error code from</param> + public static implicit operator int(ERRNO errorVal) => (int)errorVal.ErrorCode; + /// <summary> + /// C style boolean conversion. false if 0, true otherwise + /// </summary> + /// <param name="errorVal"></param> + public static implicit operator bool(ERRNO errorVal) => errorVal != 0; + /// <summary> + /// Creates a new <see cref="IntPtr"/> from the value if the stored (nint) error code + /// </summary> + /// <param name="errno">The <see cref="ERRNO"/> contating the pointer value</param> + public static implicit operator IntPtr(ERRNO errno) => new(errno.ErrorCode); + /// <summary> + /// Creates a new <c>nint</c> from the value if the stored error code + /// </summary> + /// <param name="errno">The <see cref="ERRNO"/> contating the pointer value</param> + public static implicit operator nint(ERRNO errno) => errno.ErrorCode; + + public static ERRNO operator +(ERRNO err, int add) => new(err.ErrorCode + add); + public static ERRNO operator +(ERRNO err, nint add) => new(err.ErrorCode + add); + public static ERRNO operator ++(ERRNO err) => new(err.ErrorCode + 1); + public static ERRNO operator --(ERRNO err) => new(err.ErrorCode - 1); + public static ERRNO operator -(ERRNO err, int subtract) => new(err.ErrorCode - subtract); + public static ERRNO operator -(ERRNO err, nint subtract) => new(err.ErrorCode - subtract); + + public static bool operator >(ERRNO err, ERRNO other) => err.ErrorCode > other.ErrorCode; + public static bool operator <(ERRNO err, ERRNO other) => err.ErrorCode < other.ErrorCode; + public static bool operator >=(ERRNO err, ERRNO other) => err.ErrorCode >= other.ErrorCode; + public static bool operator <=(ERRNO err, ERRNO other) => err.ErrorCode <= other.ErrorCode; + + public static bool operator >(ERRNO err, int other) => err.ErrorCode > other; + public static bool operator <(ERRNO err, int other) => err.ErrorCode < other; + public static bool operator >=(ERRNO err, int other) => err.ErrorCode >= other; + public static bool operator <=(ERRNO err, int other) => err.ErrorCode <= other; + + public static bool operator >(ERRNO err, nint other) => err.ErrorCode > other; + public static bool operator <(ERRNO err, nint other) => err.ErrorCode < other; + public static bool operator >=(ERRNO err, nint other) => err.ErrorCode >= other; + public static bool operator <=(ERRNO err, nint other) => err.ErrorCode <= other; + + public static bool operator ==(ERRNO err, ERRNO other) => err.ErrorCode == other.ErrorCode; + public static bool operator !=(ERRNO err, ERRNO other) => err.ErrorCode != other.ErrorCode; + public static bool operator ==(ERRNO err, int other) => err.ErrorCode == other; + public static bool operator !=(ERRNO err, int other) => err.ErrorCode != other; + public static bool operator ==(ERRNO err, nint other) => err.ErrorCode == other; + public static bool operator !=(ERRNO err, nint other) => err.ErrorCode != other; + + public readonly bool Equals(ERRNO other) => ErrorCode == other.ErrorCode; + public readonly override bool Equals(object obj) => obj is ERRNO other && Equals(other); + public readonly override int GetHashCode() => ErrorCode.GetHashCode(); + + /// <summary> + /// The integer error value of the current instance in radix 10 + /// </summary> + /// <returns></returns> + public readonly override string ToString() + { + //Return the string of the error code number + return ErrorCode.ToString(); + } + public readonly string ToString(string format) + { + //Return the string of the error code number + return ErrorCode.ToString(format); + } + + ///<inheritdoc/> + public readonly bool TryFormat(Span<char> destination, out int charsWritten, ReadOnlySpan<char> format, IFormatProvider provider) + { + return ErrorCode.TryFormat(destination, out charsWritten, format, provider); + } + ///<inheritdoc/> + public readonly string ToString(string format, IFormatProvider formatProvider) + { + return ErrorCode.ToString(format, formatProvider); + } + } +} diff --git a/Utils/src/Extensions/CacheExtensions.cs b/Utils/src/Extensions/CacheExtensions.cs new file mode 100644 index 0000000..5485c2d --- /dev/null +++ b/Utils/src/Extensions/CacheExtensions.cs @@ -0,0 +1,348 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: CacheExtensions.cs +* +* CacheExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Linq; +using System.Collections.Generic; + +using VNLib.Utils.Memory.Caching; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// Cache collection extensions + /// </summary> + public static class CacheExtensions + { + /// <summary> + /// <para> + /// Stores a new record. If an old record exists, the records are compared, + /// if they are not equal, the old record is evicted and the new record is stored + /// </para> + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T">A cachable object</typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <param name="record">The record to store</param> + /// <remarks> + /// Locks on the store parameter to provide mutual exclusion for non thread-safe + /// data structures. + /// </remarks> + public static void StoreRecord<TKey, T>(this IDictionary<TKey, T> store, TKey key, T record) where T : ICacheable + { + T ?oldRecord = default; + lock (store) + { + //See if an old record exists + if (!store.Remove(key, out oldRecord) || oldRecord == null) + { + //Old record doesnt exist, store and return + store[key] = record; + return; + } + //See if the old and new records and the same record + if (oldRecord.Equals(record)) + { + //records are equal, so we can exit + return; + } + //Old record is not equal, so we can store the new record and evict the old on + store[key] = record; + } + //Call evict on the old record + oldRecord.Evicted(); + } + /// <summary> + /// <para> + /// Stores a new record and updates the expiration date. If an old record exists, the records + /// are compared, if they are not equal, the old record is evicted and the new record is stored + /// </para> + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T">A cachable object</typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <param name="record">The record to store</param> + /// <param name="validFor">The new expiration time of the record</param> + /// <remarks> + /// Locks on the store parameter to provide mutual exclusion for non thread-safe + /// data structures. + /// </remarks> + public static void StoreRecord<TKey, T>(this IDictionary<TKey, T> store, TKey key, T record, TimeSpan validFor) where T : ICacheable + { + //Update the expiration time + record.Expires = DateTime.UtcNow.Add(validFor); + //Store + StoreRecord(store, key, record); + } + /// <summary> + /// <para> + /// Returns a stored record if it exists and is not expired. If the record exists + /// but has expired, it is evicted. + /// </para> + /// <para> + /// If a record is evicted, the return value evaluates to -1 and the value parameter + /// is set to the old record if the caller wished to inspect the record after the + /// eviction method completes + /// </para> + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T">A cachable object</typeparam> + /// <param name="store"></param> + /// <param name="key"></param> + /// <param name="value">The record</param> + /// <returns> + /// Gets a value indicating the reults of the operation. 0 if the record is not found, -1 if expired, 1 if + /// record is valid + /// </returns> + /// <remarks> + /// Locks on the store parameter to provide mutual exclusion for non thread-safe + /// data structures. + /// </remarks> + public static ERRNO TryGetOrEvictRecord<TKey, T>(this IDictionary<TKey, T> store, TKey key, out T? value) where T : ICacheable + { + value = default; + //Cache current date time before entering the lock + DateTime now = DateTime.UtcNow; + //Get value + lock (store) + { + //try to get the value + if (!store.TryGetValue(key, out value)) + { + //not found + return 0; + } + //Not expired + if (value.Expires > now) + { + return true; + } + //Remove from store + _ = store.Remove(key); + } + //Call the evict func + value.Evicted(); + return -1; + } + /// <summary> + /// Updates the expiration date on a record to the specified time if it exists, regardless + /// of its validity + /// </summary> + /// <typeparam name="TKey">Diction key type</typeparam> + /// <typeparam name="T">A cachable object</typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record to update</param> + /// <param name="extendedTime">The expiration time (time added to <see cref="DateTime.UtcNow"/>)</param> + /// <remarks> + /// Locks on the store parameter to provide mutual exclusion for non thread-safe + /// data structures. + /// </remarks> + public static void UpdateRecord<TKey, T>(this IDictionary<TKey, T> store, TKey key, TimeSpan extendedTime) where T : ICacheable + { + //Cacl the expiration time + DateTime expiration = DateTime.UtcNow.Add(extendedTime); + lock (store) + { + //Update the expiration time if the record exists + if (store.TryGetValue(key, out T? record) && record != null) + { + record.Expires = expiration; + } + } + } + /// <summary> + /// Evicts a stored record from the store. If the record is found, the eviction + /// method is executed + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <returns>True if the record was found and evicted</returns> + public static bool EvictRecord<TKey, T>(this IDictionary<TKey, T> store, TKey key) where T : ICacheable + { + T? record = default; + lock (store) + { + //Try to remove the record + if (!store.Remove(key, out record) || record == null) + { + //No record found or null + return false; + } + } + //Call eviction mode + record.Evicted(); + return true; + } + /// <summary> + /// Evicts all expired records from the store + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + public static void CollectRecords<TKey, T>(this IDictionary<TKey, T> store) where T : ICacheable + { + CollectRecords(store, DateTime.UtcNow); + } + + /// <summary> + /// Evicts all expired records from the store + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + /// <param name="store"></param> + /// <param name="validAfter">A time that specifies the time which expired records should be evicted</param> + public static void CollectRecords<TKey, T>(this IDictionary<TKey, T> store, DateTime validAfter) where T : ICacheable + { + //Build a query to get the keys that belong to the expired records + IEnumerable<KeyValuePair<TKey, T>> expired = store.Where(s => s.Value.Expires < validAfter); + //temp list for expired records + IEnumerable<T> evicted; + //Take lock on store + lock (store) + { + KeyValuePair<TKey, T>[] kvp = expired.ToArray(); + //enumerate to array so values can be removed while the lock is being held + foreach (KeyValuePair<TKey, T> pair in kvp) + { + //remove the record and call the eviction method + _ = store.Remove(pair); + } + //select values while lock held + evicted = kvp.Select(static v => v.Value); + } + //Iterrate over evicted records and call evicted method + foreach (T ev in evicted) + { + ev.Evicted(); + } + } + + /// <summary> + /// Allows for mutually exclusive use of a <see cref="ICacheable"/> record with a + /// state parameter + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + /// <typeparam name="State"></typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <param name="state">A user-token type state parameter to pass to the use callback method</param> + /// <param name="useCtx">A callback method that will be passed the record to use within an exclusive context</param> + public static void UseRecord<TKey, T, State>(this IDictionary<TKey, T> store, TKey key, State state, Action<T, State> useCtx) where T: ICacheable + { + lock (store) + { + //If the record exists + if(store.TryGetValue(key, out T record)) + { + //Use it within the lock statement + useCtx(record, state); + } + } + } + /// <summary> + /// Allows for mutually exclusive use of a <see cref="ICacheable"/> + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <param name="useCtx">A callback method that will be passed the record to use within an exclusive context</param> + public static void UseRecord<TKey, T>(this IDictionary<TKey, T> store, TKey key, Action<T> useCtx) where T : ICacheable + { + lock (store) + { + //If the record exists + if (store.TryGetValue(key, out T record)) + { + //Use it within the lock statement + useCtx(record); + } + } + } + /// <summary> + /// Allows for mutually exclusive use of a <see cref="ICacheable"/> record with a + /// state parameter, only if the found record is valid + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + /// <typeparam name="State"></typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <param name="state">A user-token type state parameter to pass to the use callback method</param> + /// <param name="useCtx">A callback method that will be passed the record to use within an exclusive context</param> + /// <remarks>If the record is found, but is expired, the record is evicted from the store. The callback is never invoked</remarks> + public static void UseIfValid<TKey, T, State>(this IDictionary<TKey, T> store, TKey key, State state, Action<T, State> useCtx) where T : ICacheable + { + DateTime now = DateTime.UtcNow; + T? record; + lock (store) + { + //If the record exists, check if its valid + if (store.TryGetValue(key, out record) && record.Expires < now) + { + //Use it within the lock statement + useCtx(record, state); + return; + } + //Record is no longer valid + _ = store.Remove(key); + } + //Call evicted method + record?.Evicted(); + } + /// <summary> + /// Allows for mutually exclusive use of a <see cref="ICacheable"/> record with a + /// state parameter, only if the found record is valid + /// </summary> + /// <typeparam name="TKey"></typeparam> + /// <typeparam name="T"></typeparam> + /// <param name="store"></param> + /// <param name="key">The unique key identifying the record</param> + /// <param name="useCtx">A callback method that will be passed the record to use within an exclusive context</param> + /// <remarks>If the record is found, but is expired, the record is evicted from the store. The callback is never invoked</remarks> + public static void UseIfValid<TKey, T>(this IDictionary<TKey, T> store, TKey key, Action<T> useCtx) where T : ICacheable + { + DateTime now = DateTime.UtcNow; + T? record; + lock (store) + { + //If the record exists, check if its valid + if (store.TryGetValue(key, out record) && record.Expires < now) + { + //Use it within the lock statement + useCtx(record); + return; + } + //Record is no longer valid + _ = store.Remove(key); + } + //Call evicted method + record?.Evicted(); + } + } +} diff --git a/Utils/src/Extensions/CollectionExtensions.cs b/Utils/src/Extensions/CollectionExtensions.cs new file mode 100644 index 0000000..e4ec459 --- /dev/null +++ b/Utils/src/Extensions/CollectionExtensions.cs @@ -0,0 +1,98 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: CollectionExtensions.cs +* +* CollectionExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; + +using VNLib.Utils.Memory; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// Provides collection extension methods + /// </summary> + public static class CollectionExtensions + { + /// <summary> + /// Gets a previously-stored base32 encoded value-type from the lookup and returns its initialized structure from + /// the value stored + /// </summary> + /// <typeparam name="TKey">The key type used to index the lookup</typeparam> + /// <typeparam name="TValue">An unmanaged structure type</typeparam> + /// <param name="lookup"></param> + /// <param name="key">The key used to identify the value</param> + /// <returns>The initialized structure, or default if the lookup returns null/empty string</returns> + public static TValue GetValueType<TKey, TValue>(this IIndexable<TKey, string> lookup, TKey key) where TValue : unmanaged where TKey : notnull + { + //Get value + string value = lookup[key]; + //If the string is set, recover the value and return it + return string.IsNullOrWhiteSpace(value) ? default : VnEncoding.FromBase32String<TValue>(value); + } + + /// <summary> + /// Serializes a value-type in base32 encoding and stores it at the specified key + /// </summary> + /// <typeparam name="TKey">The key type used to index the lookup</typeparam> + /// <typeparam name="TValue">An unmanaged structure type</typeparam> + /// <param name="lookup"></param> + /// <param name="key">The key used to identify the value</param> + /// <param name="value">The value to serialze</param> + public static void SetValueType<TKey, TValue>(this IIndexable<TKey, string> lookup, TKey key, TValue value) where TValue : unmanaged where TKey : notnull + { + //encode string from value type and store in lookup + lookup[key] = VnEncoding.ToBase32String(value); + } + /// <summary> + /// Executes a handler delegate on every element of the list within a try-catch block + /// and rethrows exceptions as an <see cref="AggregateException"/> + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="list"></param> + /// <param name="handler">An <see cref="Action"/> handler delegate to complete some operation on the elements within the list</param> + /// <exception cref="AggregateException"></exception> + public static void TryForeach<T>(this IEnumerable<T> list, Action<T> handler) + { + List<Exception>? exceptionList = null; + foreach(T item in list) + { + try + { + handler(item); + } + catch(Exception ex) + { + //Init new list and add the exception + exceptionList ??= new(); + exceptionList.Add(ex); + } + } + //Raise aggregate exception for all caught exceptions + if(exceptionList?.Count > 0) + { + throw new AggregateException(exceptionList); + } + } + } +} diff --git a/Utils/src/Extensions/IoExtensions.cs b/Utils/src/Extensions/IoExtensions.cs new file mode 100644 index 0000000..f312203 --- /dev/null +++ b/Utils/src/Extensions/IoExtensions.cs @@ -0,0 +1,345 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IoExtensions.cs +* +* IoExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; +using System.Runtime.Versioning; +using System.Runtime.CompilerServices; + +using VNLib.Utils.IO; +using VNLib.Utils.Memory; + +using static VNLib.Utils.Memory.Memory; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// Provieds extension methods for common IO operations + /// </summary> + public static class IoExtensions + { + /// <summary> + /// Unlocks the entire file + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + [UnsupportedOSPlatform("ios")] + [UnsupportedOSPlatform("macos")] + [UnsupportedOSPlatform("tvos")] + public static void Unlock(this FileStream fs) + { + _ = fs ?? throw new ArgumentNullException(nameof(fs)); + //Unlock the entire file + fs.Unlock(0, fs.Length); + } + + /// <summary> + /// Locks the entire file + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + [UnsupportedOSPlatform("ios")] + [UnsupportedOSPlatform("macos")] + [UnsupportedOSPlatform("tvos")] + public static void Lock(this FileStream fs) + { + _ = fs ?? throw new ArgumentNullException(nameof(fs)); + //Lock the entire length of the file + fs.Lock(0, fs.Length); + } + + /// <summary> + /// Provides an async wrapper for copying data from the current stream to another using an unmanged + /// buffer. + /// </summary> + /// <param name="source"></param> + /// <param name="dest">The destination data stream to write data to</param> + /// <param name="bufferSize">The size of the buffer to use while copying data. (Value will be clamped to the size of the stream if seeking is available)</param> + /// <param name="heap">The <see cref="IUnmangedHeap"/> to allocate the buffer from</param> + /// <param name="token">A token that may cancel asynchronous operations</param> + /// <returns>A <see cref="ValueTask"/> that completes when the copy operation has completed</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="ArgumentException"></exception> + public static async ValueTask CopyToAsync(this Stream source, Stream dest, int bufferSize, IUnmangedHeap heap, CancellationToken token = default) + { + if (source.CanSeek) + { + bufferSize = (int)Math.Min(source.Length, bufferSize); + } + //Alloc a buffer + using IMemoryOwner<byte> buffer = heap.DirectAlloc<byte>(bufferSize); + //Wait for copy to complete + await CopyToAsync(source, dest, buffer.Memory, token); + } + /// <summary> + /// Provides an async wrapper for copying data from the current stream to another with a + /// buffer from the <paramref name="heap"/> + /// </summary> + /// <param name="source"></param> + /// <param name="dest">The destination data stream to write data to</param> + /// <param name="bufferSize">The size of the buffer to use while copying data. (Value will be clamped to the size of the stream if seeking is available)</param> + /// <param name="count">The number of bytes to copy from the current stream to destination stream</param> + /// <param name="heap">The heap to alloc buffer from</param> + /// <param name="token">A token that may cancel asynchronous operations</param> + /// <returns>A <see cref="ValueTask"/> that completes when the copy operation has completed</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="ArgumentException"></exception> + public static async ValueTask CopyToAsync(this Stream source, Stream dest, long count, int bufferSize, IUnmangedHeap heap, CancellationToken token = default) + { + if (source.CanSeek) + { + bufferSize = (int)Math.Min(source.Length, bufferSize); + } + //Alloc a buffer + using IMemoryOwner<byte> buffer = heap.DirectAlloc<byte>(bufferSize); + //Wait for copy to complete + await CopyToAsync(source, dest, buffer.Memory, count, token); + } + + /// <summary> + /// Copies data from one stream to another, using self managed buffers. May allocate up to 2MB. + /// </summary> + /// <param name="source">Source stream to read from</param> + /// <param name="dest">Destination stream to write data to</param> + /// <param name="heap">The heap to allocate buffers from</param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + public static void CopyTo(this Stream source, Stream dest, IUnmangedHeap? heap = null) + { + if (!source.CanRead) + { + throw new ArgumentException("Source stream is unreadable", nameof(source)); + } + if (!dest.CanWrite) + { + throw new ArgumentException("Destination stream is unwritable", nameof(dest)); + } + heap ??= Shared; + //Get a buffer size, maximum of 2mb buffer size if the stream supports seeking, otherwise, min buf size + int bufSize = source.CanSeek ? (int)Math.Min(source.Length, MAX_BUF_SIZE) : MIN_BUF_SIZE; + //Length must be 0, so return + if (bufSize == 0) + { + return; + } + //Alloc a buffer + using UnsafeMemoryHandle<byte> buffer = heap.UnsafeAlloc<byte>(bufSize); + int read; + do + { + //read + read = source.Read(buffer.Span); + //Guard + if (read == 0) + { + break; + } + //write only the data that was read (slice) + dest.Write(buffer.Span[..read]); + } while (true); + } + /// <summary> + /// Copies data from one stream to another, using self managed buffers. May allocate up to 2MB. + /// </summary> + /// <param name="source">Source stream to read from</param> + /// <param name="dest">Destination stream to write data to</param> + /// <param name="count">Number of bytes to read/write</param> + /// <param name="heap">The heap to allocate buffers from</param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + public static void CopyTo(this Stream source, Stream dest, long count, IUnmangedHeap? heap = null) + { + if (!source.CanRead) + { + throw new ArgumentException("Source stream is unreadable", nameof(source)); + } + if (!dest.CanWrite) + { + throw new ArgumentException("Destination stream is unwritable", nameof(dest)); + } + //Set default heap + heap ??= Shared; + //Get a buffer size, maximum of 2mb buffer size if the stream supports seeking, otherwise, min buf size + int bufSize = source.CanSeek ? (int)Math.Min(source.Length, MAX_BUF_SIZE) : MIN_BUF_SIZE; + //Length must be 0, so return + if (bufSize == 0) + { + return; + } + //Alloc a buffer + using UnsafeMemoryHandle<byte> buffer = heap.UnsafeAlloc<byte>(bufSize); + //wrapper around offset pointer + long total = 0; + int read; + do + { + Span<byte> wrapper = buffer.Span[..(int)Math.Min(bufSize, (count - total))]; + //read + read = source.Read(wrapper); + //Guard + if (read == 0) + { + break; + } + //write only the data that was read (slice) + dest.Write(wrapper[..read]); + //Update total + total += read; + } while (true); + } + + /// <summary> + /// Copies data from the current stream to the destination stream using the supplied memory buffer + /// </summary> + /// <param name="source"></param> + /// <param name="dest">The destination data stream to write data to</param> + /// <param name="buffer">The buffer to use when copying data</param> + /// <param name="token">A token that may cancel asynchronous operations</param> + /// <returns>A <see cref="ValueTask"/> that completes when the copy operation has completed</returns> + /// <exception cref="ArgumentException"></exception> + public static async ValueTask CopyToAsync(this Stream source, Stream dest, Memory<byte> buffer, CancellationToken token = default) + { + //Make sure source can be read from, and dest can be written to + if (!source.CanRead) + { + throw new ArgumentException("Source stream is unreadable", nameof(source)); + } + if (!dest.CanWrite) + { + throw new ArgumentException("Destination stream is unwritable", nameof(dest)); + } + //Read in loop + int read; + while (true) + { + //read + read = await source.ReadAsync(buffer, token); + //Guard + if (read == 0) + { + break; + } + //write only the data that was read (slice) + await dest.WriteAsync(buffer[..read], token); + } + } + + /// <summary> + /// Copies data from the current stream to the destination stream using the supplied memory buffer + /// </summary> + /// <param name="source"></param> + /// <param name="dest">The destination data stream to write data to</param> + /// <param name="buffer">The buffer to use when copying data</param> + /// <param name="count">The number of bytes to copy from the current stream to destination stream</param> + /// <param name="token">A token that may cancel asynchronous operations</param> + /// <returns>A <see cref="ValueTask"/> that completes when the copy operation has completed</returns> + /// <exception cref="ArgumentException"></exception> + public static async ValueTask CopyToAsync(this Stream source, Stream dest, Memory<byte> buffer, long count, CancellationToken token = default) + { + //Make sure source can be read from, and dest can be written to + if (!source.CanRead) + { + throw new ArgumentException("Source stream is unreadable", nameof(source)); + } + if (!dest.CanWrite) + { + throw new ArgumentException("Destination stream is unwritable", nameof(dest)); + } + /* + * Track total count so we copy the exect number of + * bytes from the source + */ + long total = 0; + int bufferSize = buffer.Length; + int read; + while (true) + { + //get offset wrapper of the total buffer or remaining count + Memory<byte> offset = buffer[..(int)Math.Min(bufferSize, count - total)]; + //read + read = await source.ReadAsync(offset, token); + //Guard + if (read == 0) + { + break; + } + //write only the data that was read (slice) + await dest.WriteAsync(offset[..read], token); + //Update total + total += read; + } + } + + /// <summary> + /// Opens a file within the current directory + /// </summary> + /// <param name="dir"></param> + /// <param name="fileName">The name of the file to open</param> + /// <param name="mode">The <see cref="FileMode"/> to open the file with</param> + /// <param name="access">The <see cref="FileAccess"/> to open the file with</param> + /// <param name="share"></param> + /// <param name="bufferSize">The size of the buffer to read/write with</param> + /// <param name="options"></param> + /// <returns>The <see cref="FileStream"/> of the opened file</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static FileStream OpenFile(this DirectoryInfo dir, + string fileName, + FileMode mode, + FileAccess access, + FileShare share = FileShare.None, + int bufferSize = 4096, + FileOptions options = FileOptions.None) + { + _ = dir ?? throw new ArgumentNullException(nameof(dir)); + string fullPath = Path.Combine(dir.FullName, fileName); + return new FileStream(fullPath, mode, access, share, bufferSize, options); + } + /// <summary> + /// Deletes the speicifed file from the current directory + /// </summary> + /// <param name="dir"></param> + /// <param name="fileName">The name of the file to delete</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void DeleteFile(this DirectoryInfo dir, string fileName) + { + _ = dir ?? throw new ArgumentNullException(nameof(dir)); + string fullPath = Path.Combine(dir.FullName, fileName); + File.Delete(fullPath); + } + /// <summary> + /// Determines if a file exists within the current directory + /// </summary> + /// <param name="dir"></param> + /// <param name="fileName">The name of the file to search for</param> + /// <returns>True if the file is found and the user has permission to access the file, false otherwise</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool FileExists(this DirectoryInfo dir, string fileName) + { + _ = dir ?? throw new ArgumentNullException(nameof(dir)); + string fullPath = Path.Combine(dir.FullName, fileName); + return FileOperations.FileExists(fullPath); + } + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/JsonExtensions.cs b/Utils/src/Extensions/JsonExtensions.cs new file mode 100644 index 0000000..a27dcc0 --- /dev/null +++ b/Utils/src/Extensions/JsonExtensions.cs @@ -0,0 +1,215 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: JsonExtensions.cs +* +* JsonExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; +using System.Collections.Generic; + +using VNLib.Utils.IO; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// Specifies how to parse a timespan value from a <see cref="JsonDocument"/> element + /// </summary> + public enum TimeParseType + { + Milliseconds, + Seconds, + Minutes, + Hours, + Days, + Ticks + } + + public static class JsonExtensions + { + /// <summary> + /// Converts a JSON encoded string to an object of the specified type + /// </summary> + /// <typeparam name="T">Output type of the object</typeparam> + /// <param name="value"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during de-serialization</param> + /// <returns>The new object or default if the string is null or empty</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static T? AsJsonObject<T>(this string value, JsonSerializerOptions? options = null) + { + return !string.IsNullOrWhiteSpace(value) ? JsonSerializer.Deserialize<T>(value, options) : default; + } + /// <summary> + /// Converts a JSON encoded binary data to an object of the specified type + /// </summary> + /// <typeparam name="T">Output type of the object</typeparam> + /// <param name="utf8bin"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during de-serialization</param> + /// <returns>The new object or default if the string is null or empty</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static T? AsJsonObject<T>(this in ReadOnlySpan<byte> utf8bin, JsonSerializerOptions? options = null) + { + return utf8bin.IsEmpty ? default : JsonSerializer.Deserialize<T>(utf8bin, options); + } + /// <summary> + /// Converts a JSON encoded binary data to an object of the specified type + /// </summary> + /// <typeparam name="T">Output type of the object</typeparam> + /// <param name="utf8bin"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during de-serialization</param> + /// <returns>The new object or default if the string is null or empty</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static T? AsJsonObject<T>(this in ReadOnlyMemory<byte> utf8bin, JsonSerializerOptions? options = null) + { + return utf8bin.IsEmpty ? default : JsonSerializer.Deserialize<T>(utf8bin.Span, options); + } + /// <summary> + /// Converts a JSON encoded binary data to an object of the specified type + /// </summary> + /// <typeparam name="T">Output type of the object</typeparam> + /// <param name="utf8bin"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during de-serialization</param> + /// <returns>The new object or default if the string is null or empty</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static T? AsJsonObject<T>(this byte[] utf8bin, JsonSerializerOptions? options = null) + { + return utf8bin == null ? default : JsonSerializer.Deserialize<T>(utf8bin.AsSpan(), options); + } + /// <summary> + /// Parses a json encoded string to a json documen + /// </summary> + /// <param name="jsonString"></param> + /// <param name="options"></param> + /// <returns>If the json string is null, returns null, otherwise the json document around the data</returns> + /// <exception cref="JsonException"></exception> + public static JsonDocument? AsJsonDocument(this string jsonString, JsonDocumentOptions options = default) + { + return jsonString == null ? null : JsonDocument.Parse(jsonString, options); + } + /// <summary> + /// Shortcut extension to <see cref="JsonElement.GetProperty(string)"/> and returns a string + /// </summary> + /// <param name="element"></param> + /// <param name="propertyName">The name of the property to get the string value of</param> + /// <returns>If the property exists, returns the string stored at that property</returns> + public static string? GetPropString(this in JsonElement element, string propertyName) + { + return element.TryGetProperty(propertyName, out JsonElement el) ? el.GetString() : null; + } + /// <summary> + /// Shortcut extension to <see cref="JsonElement.GetProperty(string)"/> and returns a string + /// </summary> + /// <param name="conf"></param> + /// <param name="propertyName">The name of the property to get the string value of</param> + /// <returns>If the property exists, returns the string stored at that property</returns> + public static string? GetPropString(this IReadOnlyDictionary<string, JsonElement> conf, string propertyName) + { + return conf.TryGetValue(propertyName, out JsonElement el) ? el.GetString() : null; + } + + /// <summary> + /// Shortcut extension to <see cref="JsonElement.GetProperty(string)"/> and returns a string + /// </summary> + /// <param name="conf"></param> + /// <param name="propertyName">The name of the property to get the string value of</param> + /// <returns>If the property exists, returns the string stored at that property</returns> + public static string? GetPropString(this IDictionary<string, JsonElement> conf, string propertyName) + { + return conf.TryGetValue(propertyName, out JsonElement el) ? el.GetString() : null; + } + + /// <summary> + /// Attemts to serialze an object to a JSON encoded string + /// </summary> + /// <param name="obj"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> to use during serialization</param> + /// <returns>A JSON encoded string of the serialized object, or null if the object is null</returns> + /// <exception cref="NotSupportedException"></exception> + public static string? ToJsonString<T>(this T obj, JsonSerializerOptions? options = null) + { + return obj == null ? null : JsonSerializer.Serialize(obj, options); + } + + /// <summary> + /// Merges the current <see cref="JsonDocument"/> with another <see cref="JsonDocument"/> to + /// create a new document of combined properties + /// </summary> + /// <param name="initial"></param> + /// <param name="other">The <see cref="JsonDocument"/> to combine with the first document</param> + /// <param name="initalName">The name of the new element containing the initial document data</param> + /// <param name="secondName">The name of the new element containing the additional document data</param> + /// <returns>A new document with a parent root containing the combined objects</returns> + public static JsonDocument Merge(this JsonDocument initial, JsonDocument other, string initalName, string secondName) + { + //Open a new memory buffer + using VnMemoryStream ms = new(); + //Encapuslate the memory stream in a writer + using (Utf8JsonWriter writer = new(ms)) + { + //Write the starting + writer.WriteStartObject(); + //Write the first object name + writer.WritePropertyName(initalName); + //Write the inital docuemnt to the stream + initial.WriteTo(writer); + //Write the second object property + writer.WritePropertyName(secondName); + //Write the merging document to the stream + other.WriteTo(writer); + //End the parent element + writer.WriteEndObject(); + } + //rewind the buffer + _ = ms.Seek(0, System.IO.SeekOrigin.Begin); + //Parse the stream into the new document and return it + return JsonDocument.Parse(ms); + } + + /// <summary> + /// Parses a number value into a <see cref="TimeSpan"/> of the specified time + /// </summary> + /// <param name="el"></param> + /// <param name="type">The <see cref="TimeParseType"/> the value represents</param> + /// <returns>The <see cref="TimeSpan"/> of the value</returns> + /// <exception cref="FormatException"></exception> + /// <exception cref="OverflowException"></exception> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public static TimeSpan GetTimeSpan(this in JsonElement el, TimeParseType type) + { + return type switch + { + TimeParseType.Milliseconds => TimeSpan.FromMilliseconds(el.GetDouble()), + TimeParseType.Seconds => TimeSpan.FromSeconds(el.GetDouble()), + TimeParseType.Minutes => TimeSpan.FromMinutes(el.GetDouble()), + TimeParseType.Hours => TimeSpan.FromHours(el.GetDouble()), + TimeParseType.Days => TimeSpan.FromDays(el.GetDouble()), + TimeParseType.Ticks => TimeSpan.FromTicks(el.GetInt64()), + _ => throw new NotSupportedException(), + }; + } + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/MemoryExtensions.cs b/Utils/src/Extensions/MemoryExtensions.cs new file mode 100644 index 0000000..c8ee5ef --- /dev/null +++ b/Utils/src/Extensions/MemoryExtensions.cs @@ -0,0 +1,769 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: MemoryExtensions.cs +* +* MemoryExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text; +using System.Buffers; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +namespace VNLib.Utils.Extensions +{ + using Utils.Memory; + using VNLib.Utils.Resources; + + /// <summary> + /// Provides memory based extensions to .NET and VNLib memory abstractions + /// </summary> + public static class MemoryExtensions + { + /// <summary> + /// Rents a new array and stores it as a resource within an <see cref="OpenResourceHandle{T}"/> to return the + /// array when work is completed + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="pool"></param> + /// <param name="size">The minimum size array to allocate</param> + /// <param name="zero">Should elements from 0 to size be set to default(T)</param> + /// <returns>A new <see cref="OpenResourceHandle{T}"/> encapsulating the rented array</returns> + public static UnsafeMemoryHandle<T> Lease<T>(this ArrayPool<T> pool, int size, bool zero = false) where T: unmanaged + { + //Pool buffer handles are considered "safe" so im reusing code for now + return new(pool, size, zero); + } + + /// <summary> + /// Retreives a buffer that is at least the reqested length, and clears the array from 0-size. + /// <br></br> + /// The array may be larger than the requested size, and the entire buffer is zeroed + /// </summary> + /// <param name="pool"></param> + /// <param name="size">The minimum length of the array</param> + /// <param name="zero">True if contents should be zeroed</param> + /// <returns>The zeroed array</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static T[] Rent<T>(this ArrayPool<T> pool, int size, bool zero) + { + //Rent the array + T[] arr = pool.Rent(size); + //If zero flag is set, zero only the used section + if (zero) + { + Array.Fill(arr, default); + } + return arr; + } + + /// <summary> + /// Copies the characters within the memory handle to a <see cref="string"/> + /// </summary> + /// <returns>The string representation of the buffer</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static string ToString<T>(this T charBuffer) where T: IMemoryHandle<char> + { + return charBuffer.Span.ToString(); + } + + /// <summary> + /// Wraps the <see cref="MemoryHandle{T}"/> instance in System.Buffers.MemoryManager + /// wrapper to provide <see cref="Memory{T}"/> buffers from umanaged handles. + /// </summary> + /// <typeparam name="T">The unmanaged data type</typeparam> + /// <param name="handle"></param> + /// <param name="ownsHandle"> + /// A value that indicates if the new <see cref="MemoryManager{T}"/> owns the handle. + /// When <c>true</c>, the new <see cref="MemoryManager{T}"/> maintains the lifetime of the handle. + /// </param> + /// <returns>The <see cref="MemoryManager{T}"/> wrapper</returns> + /// <remarks>NOTE: This wrapper now manages the lifetime of the current handle</remarks> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryManager<T> ToMemoryManager<T>(this MemoryHandle<T> handle, bool ownsHandle = true) where T : unmanaged + { + _ = handle ?? throw new ArgumentNullException(nameof(handle)); + return new SysBufferMemoryManager<T>(handle, ownsHandle); + } + + /// <summary> + /// Wraps the <see cref="VnTempBuffer{T}"/> instance in System.Buffers.MemoryManager + /// wrapper to provide <see cref="Memory{T}"/> buffers from umanaged handles. + /// </summary> + /// <typeparam name="T">The unmanaged data type</typeparam> + /// <param name="handle"></param> + /// <param name="ownsHandle"> + /// A value that indicates if the new <see cref="MemoryManager{T}"/> owns the handle. + /// When <c>true</c>, the new <see cref="MemoryManager{T}"/> maintains the lifetime of the handle. + /// </param> + /// <returns>The <see cref="MemoryManager{T}"/> wrapper</returns> + /// <remarks>NOTE: This wrapper now manages the lifetime of the current handle</remarks> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryManager<T> ToMemoryManager<T>(this VnTempBuffer<T> handle, bool ownsHandle = true) where T : unmanaged + { + _ = handle ?? throw new ArgumentNullException(nameof(handle)); + return new SysBufferMemoryManager<T>(handle, ownsHandle); + } + + /// <summary> + /// Allows direct allocation of a fixed size <see cref="MemoryManager{T}"/> from a <see cref="PrivateHeap"/> instance + /// of the specified number of elements + /// </summary> + /// <typeparam name="T">The unmanaged data type</typeparam> + /// <param name="heap"></param> + /// <param name="size">The number of elements to allocate on the heap</param> + /// <param name="zero">Optionally zeros conents of the block when allocated</param> + /// <returns>The <see cref="MemoryManager{T}"/> wrapper around the block of memory</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryManager<T> DirectAlloc<T>(this IUnmangedHeap heap, ulong size, bool zero = false) where T : unmanaged + { + return new SysBufferMemoryManager<T>(heap, size, zero); + } + + /// <summary> + /// Allows direct allocation of a fixed size <see cref="MemoryManager{T}"/> from a <see cref="PrivateHeap"/> instance + /// of the specified number of elements + /// </summary> + /// <typeparam name="T">The unmanaged data type</typeparam> + /// <param name="heap"></param> + /// <param name="size">The number of elements to allocate on the heap</param> + /// <param name="zero">Optionally zeros conents of the block when allocated</param> + /// <returns>The <see cref="MemoryManager{T}"/> wrapper around the block of memory</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryManager<T> DirectAlloc<T>(this IUnmangedHeap heap, long size, bool zero = false) where T : unmanaged + { + return size < 0 ? throw new ArgumentOutOfRangeException(nameof(size)) : DirectAlloc<T>(heap, (ulong)size, zero); + } + /// <summary> + /// Gets an offset pointer from the base postion to the number of bytes specified. Performs bounds checks + /// </summary> + /// <param name="memory"></param> + /// <param name="elements">Number of elements of type to offset</param> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + /// <returns><typeparamref name="T"/> pointer to the memory offset specified</returns> + /// [MethodImpl(MethodImplOptions.AggressiveInlining)] + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe T* GetOffset<T>(this MemoryHandle<T> memory, long elements) where T : unmanaged + { + return elements < 0 ? throw new ArgumentOutOfRangeException(nameof(elements)) : memory.GetOffset((ulong)elements); + } + /// <summary> + /// Resizes the current handle on the heap + /// </summary> + /// <param name="memory"></param> + /// <param name="elements">Positive number of elemnts the current handle should referrence</param> + /// <exception cref="OverflowException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Resize<T>(this MemoryHandle<T> memory, long elements) where T : unmanaged + { + if (elements < 0) + { + throw new ArgumentOutOfRangeException(nameof(elements)); + } + memory.Resize((ulong)elements); + } + + /// <summary> + /// Resizes the target handle only if the handle is smaller than the requested element count + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="handle"></param> + /// <param name="count">The number of elements to resize to</param> + /// <exception cref="OverflowException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ResizeIfSmaller<T>(this MemoryHandle<T> handle, long count) where T : unmanaged + { + if(count < 0) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + ResizeIfSmaller(handle, (ulong)count); + } + + /// <summary> + /// Resizes the target handle only if the handle is smaller than the requested element count + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="handle"></param> + /// <param name="count">The number of elements to resize to</param> + /// <exception cref="OverflowException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ResizeIfSmaller<T>(this MemoryHandle<T> handle, ulong count) where T : unmanaged + { + //Check handle size + if(handle.Length < count) + { + //handle too small, resize + handle.Resize(count); + } + } + +#if TARGET_64_BIT + /// <summary> + /// Gets a 64bit friendly span offset for the current <see cref="MemoryHandle{T}"/> + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="block"></param> + /// <param name="offset">The offset (in elements) from the begining of the block</param> + /// <param name="size">The size of the block (in elements)</param> + /// <returns>The offset span</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe Span<T> GetOffsetSpan<T>(this MemoryHandle<T> block, ulong offset, int size) where T: unmanaged + { + _ = block ?? throw new ArgumentNullException(nameof(block)); + if(size < 0) + { + throw new ArgumentOutOfRangeException(nameof(size)); + } + if(size == 0) + { + return Span<T>.Empty; + } + //Make sure the offset size is within the size of the block + if(offset + (ulong)size <= block.Length) + { + //Get long offset from the destination handle + void* ofPtr = block.GetOffset(offset); + return new Span<T>(ofPtr, size); + } + throw new ArgumentOutOfRangeException(nameof(size)); + } + /// <summary> + /// Gets a 64bit friendly span offset for the current <see cref="MemoryHandle{T}"/> + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="block"></param> + /// <param name="offset">The offset (in elements) from the begining of the block</param> + /// <param name="size">The size of the block (in elements)</param> + /// <returns>The offset span</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe Span<T> GetOffsetSpan<T>(this MemoryHandle<T> block, long offset, int size) where T : unmanaged + { + return offset < 0 ? throw new ArgumentOutOfRangeException(nameof(offset)) : block.GetOffsetSpan<T>((ulong)offset, size); + } + + + /// <summary> + /// Gets a <see cref="SubSequence{T}"/> window within the current block + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="block"></param> + /// <param name="offset">An offset within the handle</param> + /// <param name="size">The size of the window</param> + /// <returns>The new <see cref="SubSequence{T}"/> within the block</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static SubSequence<T> GetSubSequence<T>(this MemoryHandle<T> block, ulong offset, int size) where T : unmanaged + { + return new SubSequence<T>(block, offset, size); + } +#else + + /// <summary> + /// Gets a <see cref="SubSequence{T}"/> window within the current block + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="block"></param> + /// <param name="offset">An offset within the handle</param> + /// <param name="size">The size of the window</param> + /// <returns>The new <see cref="SubSequence{T}"/> within the block</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static SubSequence<T> GetSubSequence<T>(this MemoryHandle<T> block, int offset, int size) where T : unmanaged + { + return new SubSequence<T>(block, offset, size); + } + + /// <summary> + /// Gets a 64bit friendly span offset for the current <see cref="MemoryHandle{T}"/> + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="block"></param> + /// <param name="offset">The offset (in elements) from the begining of the block</param> + /// <param name="size">The size of the block (in elements)</param> + /// <returns>The offset span</returns> + /// <exception cref="OverflowException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe Span<T> GetOffsetSpan<T>(this MemoryHandle<T> block, long offset, int size) where T : unmanaged + { + //TODO fix 32bit/64 bit, this is a safe lazy workaround + return block.Span.Slice(checked((int) offset), size); + } +#endif + + /// <summary> + /// Wraps the current instance with a <see cref="MemoryPool{T}"/> wrapper + /// to allow System.Memory buffer rentals. + /// </summary> + /// <typeparam name="T">The unmanged data type to provide allocations from</typeparam> + /// <returns>The new <see cref="MemoryPool{T}"/> heap wrapper.</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryPool<T> ToPool<T>(this IUnmangedHeap heap) where T : unmanaged + { + return new PrivateBuffersMemoryPool<T>(heap); + } + + /// <summary> + /// Allocates a structure of the specified type on the current unmanged heap and zero's its memory + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="heap"></param> + /// <returns>A pointer to the structure ready for use.</returns> + /// <remarks>Allocations must be freed with <see cref="StructFree{T}(IUnmangedHeap, T*)"/></remarks> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe T* StructAlloc<T>(this IUnmangedHeap heap) where T : unmanaged + { + //Allocate the struct on the heap and zero memory it points to + IntPtr handle = heap.Alloc(1, (uint)sizeof(T), true); + //returns the handle + return (T*)handle; + } + /// <summary> + /// Frees a structure at the specified address from the this heap. + /// This must be the same heap the structure was allocated from + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="heap"></param> + /// <param name="structPtr">A pointer to the structure</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe void StructFree<T>(this IUnmangedHeap heap, T* structPtr) where T : unmanaged + { + IntPtr block = new(structPtr); + //Free block from heap + heap.Free(ref block); + //Clear ref + *structPtr = default; + } + /// <summary> + /// Allocates a block of unmanaged memory of the number of elements to store of an unmanged type + /// </summary> + /// <typeparam name="T">Unmanaged data type to create a block of</typeparam> + /// <param name="heap"></param> + /// <param name="elements">The size of the block (number of elements)</param> + /// <param name="zero">A flag that zeros the allocated block before returned</param> + /// <returns>The unmanaged <see cref="MemoryHandle{T}"/></returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe MemoryHandle<T> Alloc<T>(this IUnmangedHeap heap, ulong elements, bool zero = false) where T : unmanaged + { + //Minimum of one element + elements = Math.Max(elements, 1); + //Get element size + uint elementSize = (uint)sizeof(T); + //If zero flag is set then specify zeroing memory + IntPtr block = heap.Alloc(elements, elementSize, zero); + //Return handle wrapper + return new MemoryHandle<T>(heap, block, elements, zero); + } + /// <summary> + /// Allocates a block of unmanaged memory of the number of elements to store of an unmanged type + /// </summary> + /// <typeparam name="T">Unmanaged data type to create a block of</typeparam> + /// <param name="heap"></param> + /// <param name="elements">The size of the block (number of elements)</param> + /// <param name="zero">A flag that zeros the allocated block before returned</param> + /// <returns>The unmanaged <see cref="MemoryHandle{T}"/></returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryHandle<T> Alloc<T>(this IUnmangedHeap heap, long elements, bool zero = false) where T : unmanaged + { + return elements < 0 ? throw new ArgumentOutOfRangeException(nameof(elements)) : Alloc<T>(heap, (ulong)elements, zero); + } + /// <summary> + /// Allocates a buffer from the current heap and initialzies it by copying the initial data buffer + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="heap"></param> + /// <param name="initialData">The initial data to set the buffer to</param> + /// <returns>The initalized <see cref="MemoryHandle{T}"/> block</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static MemoryHandle<T> AllocAndCopy<T>(this IUnmangedHeap heap, ReadOnlySpan<T> initialData) where T:unmanaged + { + MemoryHandle<T> handle = heap.Alloc<T>(initialData.Length); + Memory.Copy(initialData, handle, 0); + return handle; + } + + /// <summary> + /// Copies data from the input buffer to the current handle and resizes the handle to the + /// size of the buffer + /// </summary> + /// <typeparam name="T">The unamanged value type</typeparam> + /// <param name="handle"></param> + /// <param name="input">The input buffer to copy data from</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void WriteAndResize<T>(this MemoryHandle<T> handle, ReadOnlySpan<T> input) where T: unmanaged + { + handle.Resize(input.Length); + Memory.Copy(input, handle, 0); + } + + /// <summary> + /// Allocates a block of unamanged memory of the number of elements of an unmanaged type, and + /// returns the <see cref="UnsafeMemoryHandle{T}"/> that must be used cautiously + /// </summary> + /// <typeparam name="T">The unamanged value type</typeparam> + /// <param name="heap">The heap to allocate block from</param> + /// <param name="elements">The number of elements to allocate</param> + /// <param name="zero">A flag to zero the initial contents of the buffer</param> + /// <returns>The allocated handle of the specified number of elements</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static unsafe UnsafeMemoryHandle<T> UnsafeAlloc<T>(this IUnmangedHeap heap, int elements, bool zero = false) where T : unmanaged + { + if (elements < 1) + { + throw new ArgumentException("Elements must be greater than 0", nameof(elements)); + } + //Minimum of one element + elements = Math.Max(elements, 1); + //Get element size + uint elementSize = (uint)sizeof(T); + //If zero flag is set then specify zeroing memory + IntPtr block = heap.Alloc((uint)elements, elementSize, zero); + //handle wrapper + return new (heap, block, elements); + } + + #region VnBufferWriter + + /// <summary> + /// Formats and appends a value type to the writer with proper endianess + /// </summary> + /// <param name="buffer"></param> + /// <param name="value">The value to format and append to the buffer</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static void Append<T>(this ref ForwardOnlyWriter<byte> buffer, T value) where T: unmanaged + { + //Calc size of structure and fix te size of the buffer + int size = Unsafe.SizeOf<T>(); + Span<byte> output = buffer.Remaining[..size]; + + //Format value and write to buffer + MemoryMarshal.Write(output, ref value); + + //If byte order is reversed, reverse elements + if (!BitConverter.IsLittleEndian) + { + output.Reverse(); + } + + //Update written posiion + buffer.Advance(size); + } + + /// <summary> + /// Formats and appends a value type to the writer with proper endianess + /// </summary> + /// <param name="buffer"></param> + /// <param name="value">The value to format and append to the buffer</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static void Append<T>(this ref ForwardOnlyMemoryWriter<byte> buffer, T value) where T : struct + { + //Format value and write to buffer + int size = Unsafe.SizeOf<T>(); + Span<byte> output = buffer.Remaining.Span[..size]; + + //Format value and write to buffer + MemoryMarshal.Write(output, ref value); + + //If byte order is reversed, reverse elements + if (BitConverter.IsLittleEndian) + { + output.Reverse(); + } + + //Update written posiion + buffer.Advance(size); + } + + /// <summary> + /// Formats and appends the value to end of the buffer + /// </summary> + /// <param name="buffer"></param> + /// <param name="value">The value to format and append to the buffer</param> + /// <param name="format">An optional format argument</param> + /// <param name="formatProvider"></param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Append<T>(this ref ForwardOnlyWriter<char> buffer, T value, ReadOnlySpan<char> format = default, IFormatProvider? formatProvider = default) where T : ISpanFormattable + { + //Format value and write to buffer + if (!value.TryFormat(buffer.Remaining, out int charsWritten, format, formatProvider)) + { + throw new ArgumentOutOfRangeException(nameof(buffer), "The value could not be formatted and appended to the buffer, because there is not enough available space"); + } + //Update written posiion + buffer.Advance(charsWritten); + } + + /// <summary> + /// Formats and appends the value to end of the buffer + /// </summary> + /// <param name="buffer"></param> + /// <param name="value">The value to format and append to the buffer</param> + /// <param name="format">An optional format argument</param> + /// <param name="formatProvider"></param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Append<T>(this ref ForwardOnlyMemoryWriter<char> buffer, T value, ReadOnlySpan<char> format = default, IFormatProvider? formatProvider = default) where T : ISpanFormattable + { + //Format value and write to buffer + if (!value.TryFormat(buffer.Remaining.Span, out int charsWritten, format, formatProvider)) + { + throw new ArgumentOutOfRangeException(nameof(buffer), "The value could not be formatted and appended to the buffer, because there is not enough available space"); + } + //Update written posiion + buffer.Advance(charsWritten); + } + + + + /// <summary> + /// Encodes a set of characters in the input characters span and any characters + /// in the internal buffer into a sequence of bytes that are stored in the input + /// byte span. A parameter indicates whether to clear the internal state of the + /// encoder after the conversion. + /// </summary> + /// <param name="enc"></param> + /// <param name="chars">Character buffer to encode</param> + /// <param name="offset">The offset in the char buffer to begin encoding chars from</param> + /// <param name="charCount">The number of characers to encode</param> + /// <param name="writer">The buffer writer to use</param> + /// <param name="flush">true to clear the internal state of the encoder after the conversion; otherwise, false.</param> + /// <returns>The actual number of bytes written at the location indicated by the bytes parameter.</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetBytes(this Encoder enc, char[] chars, int offset, int charCount, ref ForwardOnlyWriter<byte> writer, bool flush) + { + return GetBytes(enc, chars.AsSpan(offset, charCount), ref writer, flush); + } + /// <summary> + /// Encodes a set of characters in the input characters span and any characters + /// in the internal buffer into a sequence of bytes that are stored in the input + /// byte span. A parameter indicates whether to clear the internal state of the + /// encoder after the conversion. + /// </summary> + /// <param name="enc"></param> + /// <param name="chars">The character buffer to encode</param> + /// <param name="writer">The buffer writer to use</param> + /// <param name="flush">true to clear the internal state of the encoder after the conversion; otherwise, false.</param> + /// <returns>The actual number of bytes written at the location indicated by the bytes parameter.</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetBytes(this Encoder enc, ReadOnlySpan<char> chars, ref ForwardOnlyWriter<byte> writer, bool flush) + { + //Encode the characters + int written = enc.GetBytes(chars, writer.Remaining, flush); + //Update the writer position + writer.Advance(written); + return written; + } + /// <summary> + /// Encodes a set of characters in the input characters span and any characters + /// in the internal buffer into a sequence of bytes that are stored in the input + /// byte span. + /// </summary> + /// <param name="encoding"></param> + /// <param name="chars">The character buffer to encode</param> + /// <param name="writer">The buffer writer to use</param> + /// <returns>The actual number of bytes written at the location indicated by the bytes parameter.</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static int GetBytes(this Encoding encoding, ReadOnlySpan<char> chars, ref ForwardOnlyWriter<byte> writer) + { + //Encode the characters + int written = encoding.GetBytes(chars, writer.Remaining); + //Update the writer position + writer.Advance(written); + return written; + } + /// <summary> + /// Decodes a character buffer in the input characters span and any characters + /// in the internal buffer into a sequence of bytes that are stored in the input + /// byte span. + /// </summary> + /// <param name="encoding"></param> + /// <param name="bytes">The binary buffer to decode</param> + /// <param name="writer">The buffer writer to use</param> + /// <returns>The actual number of *characters* written at the location indicated by the chars parameter.</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static int GetChars(this Encoding encoding, ReadOnlySpan<byte> bytes, ref ForwardOnlyWriter<char> writer) + { + int charCount = encoding.GetCharCount(bytes); + //Encode the characters + _ = encoding.GetChars(bytes, writer.Remaining); + //Update the writer position + writer.Advance(charCount); + return charCount; + } + + /// <summary> + /// Converts the buffer data to a <see cref="PrivateString"/> + /// </summary> + /// <returns>A <see cref="PrivateString"/> instance that owns the underlying string memory</returns> + public static PrivateString ToPrivate(this ref ForwardOnlyWriter<char> buffer) => new(buffer.ToString(), true); + /// <summary> + /// Gets a <see cref="Span{T}"/> over the modified section of the internal buffer + /// </summary> + /// <returns>A <see cref="Span{T}"/> over the modified data</returns> + public static Span<T> AsSpan<T>(this ref ForwardOnlyWriter<T> buffer) => buffer.Buffer[..buffer.Written]; + + + #endregion + + /// <summary> + /// Slices the current array by the specified starting offset to the end + /// of the array + /// </summary> + /// <typeparam name="T">The array type</typeparam> + /// <param name="arr"></param> + /// <param name="start">The start offset of the new array slice</param> + /// <returns>The sliced array</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static T[] Slice<T>(this T[] arr, int start) + { + if(start < 0 || start > arr.Length) + { + throw new ArgumentOutOfRangeException(nameof(start)); + } + Range sliceRange = new(start, arr.Length - start); + return RuntimeHelpers.GetSubArray(arr, sliceRange); + } + /// <summary> + /// Slices the current array by the specified starting offset to including the + /// speciifed number of items + /// </summary> + /// <typeparam name="T">The array type</typeparam> + /// <param name="arr"></param> + /// <param name="start">The start offset of the new array slice</param> + /// <param name="count">The size of the new array</param> + /// <returns>The sliced array</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static T[] Slice<T>(this T[] arr, int start, int count) + { + if(start < 0) + { + throw new ArgumentOutOfRangeException(nameof(start)); + } + if(count < 0) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + if(start + count >= arr.Length) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + if(count == 0) + { + return Array.Empty<T>(); + } + //Calc the slice range + Range sliceRange = new(start, start + count); + return RuntimeHelpers.GetSubArray(arr, sliceRange); + } + + /// <summary> + /// Creates a new sub-sequence over the target handle. (allows for convient sub span) + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="handle"></param> + /// <param name="start">Intial offset into the handle</param> + /// <returns>The sub-sequence of the current handle</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span<T> AsSpan<T>(this IMemoryHandle<T> handle, int start) => handle.Span[start..]; + + /// <summary> + /// Creates a new sub-sequence over the target handle. (allows for convient sub span) + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="handle"></param> + /// <param name="start">Intial offset into the handle</param> + /// <param name="count">The number of elements within the new sequence</param> + /// <returns>The sub-sequence of the current handle</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span<T> AsSpan<T>(this IMemoryHandle<T> handle, int start, int count) => handle.Span.Slice(start, count); + + /// <summary> + /// Creates a new sub-sequence over the target handle. (allows for convient sub span) + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="handle"></param> + /// <param name="start">Intial offset into the handle</param> + /// <returns>The sub-sequence of the current handle</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span<T> AsSpan<T>(this in UnsafeMemoryHandle<T> handle, int start) where T: unmanaged => handle.Span[start..]; + + /// <summary> + /// Creates a new sub-sequence over the target handle. (allows for convient sub span) + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="handle"></param> + /// <param name="start">Intial offset into the handle</param> + /// <param name="count">The number of elements within the new sequence</param> + /// <returns>The sub-sequence of the current handle</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static Span<T> AsSpan<T>(this in UnsafeMemoryHandle<T> handle, int start, int count) where T : unmanaged => handle.Span.Slice(start, count); + + /// <summary> + /// Raises an <see cref="ObjectDisposedException"/> if the current handle + /// has been disposed or set as invalid + /// </summary> + /// <param name="handle"></param> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void ThrowIfClosed(this SafeHandle handle) + { + if (handle.IsClosed || handle.IsInvalid) + { + throw new ObjectDisposedException(handle.GetType().Name); + } + } + } +} diff --git a/Utils/src/Extensions/MutexReleaser.cs b/Utils/src/Extensions/MutexReleaser.cs new file mode 100644 index 0000000..84dd60f --- /dev/null +++ b/Utils/src/Extensions/MutexReleaser.cs @@ -0,0 +1,62 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: MutexReleaser.cs +* +* MutexReleaser.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// Represents a releaser handle for a <see cref="Mutex"/> + /// that has been entered and will be released. Best if used + /// within a using() statment + /// </summary> + public readonly struct MutexReleaser : IDisposable, IEquatable<MutexReleaser> + { + private readonly Mutex _mutext; + internal MutexReleaser(Mutex mutex) => _mutext = mutex; + /// <summary> + /// Releases the held System.Threading.Mutex once. + /// </summary> + public readonly void Dispose() => _mutext.ReleaseMutex(); + /// <summary> + /// Releases the held System.Threading.Mutex once. + /// </summary> + public readonly void ReleaseMutext() => _mutext.ReleaseMutex(); + + ///<inheritdoc/> + public bool Equals(MutexReleaser other) => _mutext.Equals(other._mutext); + + ///<inheritdoc/> + public override bool Equals(object? obj) => obj is MutexReleaser releaser && Equals(releaser); + + ///<inheritdoc/> + public override int GetHashCode() => _mutext.GetHashCode(); + + ///<inheritdoc/> + public static bool operator ==(MutexReleaser left, MutexReleaser right) => left.Equals(right); + ///<inheritdoc/> + public static bool operator !=(MutexReleaser left, MutexReleaser right) => !(left == right); + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/SafeLibraryExtensions.cs b/Utils/src/Extensions/SafeLibraryExtensions.cs new file mode 100644 index 0000000..8866059 --- /dev/null +++ b/Utils/src/Extensions/SafeLibraryExtensions.cs @@ -0,0 +1,103 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SafeLibraryExtensions.cs +* +* SafeLibraryExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Reflection; + +using VNLib.Utils.Native; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// When applied to a delegate, specifies the name of the native method to load + /// </summary> + [AttributeUsage(AttributeTargets.Delegate)] + public sealed class SafeMethodNameAttribute : Attribute + { + /// <summary> + /// Creates a new <see cref="SafeMethodNameAttribute"/> + /// </summary> + /// <param name="MethodName">The name of the native method</param> + public SafeMethodNameAttribute(string MethodName) => this.MethodName = MethodName; + /// <summary> + /// Creates a new <see cref="SafeMethodNameAttribute"/>, that uses the + /// delegate name as the native method name + /// </summary> + public SafeMethodNameAttribute() => MethodName = null; + /// <summary> + /// The name of the native method + /// </summary> + public string? MethodName { get; } + } + + + /// <summary> + /// Contains native library extension methods + /// </summary> + public static class SafeLibraryExtensions + { + const string _missMemberExceptionMessage = $"The delegate type is missing the required {nameof(SafeMethodNameAttribute)} to designate the native method to load"; + + /// <summary> + /// Loads a native method from the current <see cref="SafeLibraryHandle"/> + /// that has a <see cref="SafeMethodNameAttribute"/> + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="library"></param> + /// <returns></returns> + /// <exception cref="MissingMemberException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="EntryPointNotFoundException"></exception> + public static SafeMethodHandle<T> GetMethod<T>(this SafeLibraryHandle library) where T : Delegate + { + Type t = typeof(T); + //Get the method name attribute + SafeMethodNameAttribute? attr = t.GetCustomAttribute<SafeMethodNameAttribute>(); + _ = attr ?? throw new MissingMemberException(_missMemberExceptionMessage); + return library.GetMethod<T>(attr.MethodName ?? t.Name); + } + /// <summary> + /// Loads a native method from the current <see cref="SafeLibraryHandle"/> + /// that has a <see cref="SafeMethodNameAttribute"/> + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="library"></param> + /// <returns></returns> + /// <exception cref="MissingMemberException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="EntryPointNotFoundException"></exception> + /// <remarks> + /// The libraries handle count is left unmodified + /// </remarks> + public static T DangerousGetMethod<T>(this SafeLibraryHandle library) where T: Delegate + { + Type t = typeof(T); + //Get the method name attribute + SafeMethodNameAttribute? attr = t.GetCustomAttribute<SafeMethodNameAttribute>(); + return string.IsNullOrWhiteSpace(attr?.MethodName) + ? throw new MissingMemberException(_missMemberExceptionMessage) + : library.DangerousGetMethod<T>(attr.MethodName); + } + } +} diff --git a/Utils/src/Extensions/SemSlimReleaser.cs b/Utils/src/Extensions/SemSlimReleaser.cs new file mode 100644 index 0000000..c8a22fe --- /dev/null +++ b/Utils/src/Extensions/SemSlimReleaser.cs @@ -0,0 +1,51 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SemSlimReleaser.cs +* +* SemSlimReleaser.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; + +namespace VNLib.Utils.Extensions +{ + /// <summary> + /// Represents a releaser handle for a <see cref="SemaphoreSlim"/> + /// that has been entered and will be released. Best if used + /// within a using() statment + /// </summary> + public readonly struct SemSlimReleaser : IDisposable + { + private readonly SemaphoreSlim _semaphore; + internal SemSlimReleaser(SemaphoreSlim semaphore) => _semaphore = semaphore; + /// <summary> + /// Releases the System.Threading.SemaphoreSlim object once. + /// </summary> + public readonly void Dispose() => _semaphore.Release(); + /// <summary> + /// Releases the System.Threading.SemaphoreSlim object once. + /// </summary> + /// <returns>The previous count of the <see cref="SemaphoreSlim"/></returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="SemaphoreFullException"></exception> + public readonly int Release() => _semaphore.Release(); + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/StringExtensions.cs b/Utils/src/Extensions/StringExtensions.cs new file mode 100644 index 0000000..09d6517 --- /dev/null +++ b/Utils/src/Extensions/StringExtensions.cs @@ -0,0 +1,481 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: StringExtensions.cs +* +* StringExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; + +using VNLib.Utils.Memory; + +namespace VNLib.Utils.Extensions +{ + public delegate void StatelessSpanAction(ReadOnlySpan<char> line); + + /// <summary> + /// Extention methods for string (character buffer) + /// </summary> + public static class StringExtensions + { + /// <summary> + /// Split a string based on split value and insert into the specified list + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The value to split the string on</param> + /// <param name="output">The list to output data to</param> + /// <param name="options">String split options</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split<T>(this string value, string splitter, T output, StringSplitOptions options) where T : ICollection<string> + { + Split(value, splitter.AsSpan(), output, options); + } + /// <summary> + /// Split a string based on split value and insert into the specified list + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The value to split the string on</param> + /// <param name="output">The list to output data to</param> + /// <param name="options">String split options</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split<T>(this string value, char splitter, T output, StringSplitOptions options) where T: ICollection<string> + { + //Create span from char pointer + ReadOnlySpan<char> cs = MemoryMarshal.CreateReadOnlySpan(ref splitter, 1); + //Call the split function on the span + Split(value, cs, output, options); + } + /// <summary> + /// Split a string based on split value and insert into the specified list + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The value to split the string on</param> + /// <param name="output">The list to output data to</param> + /// <param name="options">String split options</param> + /// <exception cref="ArgumentNullException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split<T>(this string value, ReadOnlySpan<char> splitter, T output, StringSplitOptions options) where T : ICollection<string> + { + Split(value.AsSpan(), splitter, output, options); + } + /// <summary> + /// Split a string based on split value and insert into the specified list + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The value to split the string on</param> + /// <param name="output">The list to output data to</param> + /// <param name="options">String split options</param> + /// <exception cref="ArgumentNullException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split<T>(this in ReadOnlySpan<char> value, char splitter, T output, StringSplitOptions options) where T : ICollection<string> + { + //Create span from char pointer + ReadOnlySpan<char> cs = MemoryMarshal.CreateReadOnlySpan(ref splitter, 1); + //Call the split function on the span + Split(in value, cs, output, options); + } + /// <summary> + /// Split a <see cref="ReadOnlySpan{T}"/> based on split value and insert into the specified list + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The value to split the string on</param> + /// <param name="output">The list to output data to</param> + /// <param name="options">String split options</param> + /// <exception cref="ArgumentNullException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split<T>(this in ReadOnlySpan<char> value, ReadOnlySpan<char> splitter, T output, StringSplitOptions options) where T : ICollection<string> + { + //Create a local function that adds the split strings to the list + static void SplitFound(ReadOnlySpan<char> split, T output) => output.Add(split.ToString()); + //Invoke the split function with the local callback method + Split(in value, splitter, options, SplitFound, output); + } + /// <summary> + /// Split a <see cref="ReadOnlySpan{T}"/> based on split value and pass it to the split delegate handler + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The sequence to split the string on</param> + /// <param name="options">String split options</param> + /// <param name="splitCb">The action to invoke when a split segment has been found</param> + /// <param name="state">The state to pass to the callback handler</param> + /// <exception cref="ArgumentNullException"></exception> + public static void Split<T>(this in ReadOnlySpan<char> value, ReadOnlySpan<char> splitter, StringSplitOptions options, ReadOnlySpanAction<char, T> splitCb, T state) + { + _ = splitCb ?? throw new ArgumentNullException(nameof(splitCb)); + //Get span over string + ForwardOnlyReader<char> reader = new(value); + //No string options + if (options == 0) + { + do + { + //Find index of the splitter + int start = reader.Window.IndexOf(splitter); + //guard + if (start == -1) + { + break; + } + //Trim and add it regardless of length + splitCb(reader.Window[..start], state); + //shift window + reader.Advance(start + splitter.Length); + } while (true); + //Trim remaining and add it regardless of length + splitCb(reader.Window, state); + } + //Trim but do not remove empties + else if ((options & StringSplitOptions.RemoveEmptyEntries) == 0) + { + do + { + //Find index of the splitter + int start = reader.Window.IndexOf(splitter); + //guard + if (start == -1) + { + break; + } + //Trim and add it regardless of length + splitCb(reader.Window[..start].Trim(), state); + //shift window + reader.Advance(start + splitter.Length); + } while (true); + //Trim remaining and add it regardless of length + splitCb(reader.Window.Trim(), state); + } + //Remove empty entires but do not trim them + else if ((options & StringSplitOptions.TrimEntries) == 0) + { + //Get data before splitter and trim it + ReadOnlySpan<char> data; + do + { + //Find index of the splitter + int start = reader.Window.IndexOf(splitter); + //guard + if (start == -1) + { + break; + } + //Get data before splitter and trim it + data = reader.Window[..start]; + //If its not empty, then add it to the list + if (!data.IsEmpty) + { + splitCb(data, state); + } + //shift window + reader.Advance(start + splitter.Length); + } while (true); + //Add if not empty + if (reader.WindowSize > 0) + { + splitCb(reader.Window, state); + } + } + //Must mean remove and trim + else + { + //Get data before splitter and trim it + ReadOnlySpan<char> data; + do + { + //Find index of the splitter + int start = reader.Window.IndexOf(splitter); + //guard + if (start == -1) + { + break; + } + //Get data before splitter and trim it + data = reader.Window[..start].Trim(); + //If its not empty, then add it to the list + if (!data.IsEmpty) + { + splitCb(data, state); + } + //shift window + reader.Advance(start + splitter.Length); + } while (true); + //Trim remaining + data = reader.Window.Trim(); + //Add if not empty + if (!data.IsEmpty) + { + splitCb(data, state); + } + } + } + + /// <summary> + /// Split a <see cref="ReadOnlySpan{T}"/> based on split value and pass it to the split delegate handler + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The character to split the string on</param> + /// <param name="options">String split options</param> + /// <param name="splitCb">The action to invoke when a split segment has been found</param> + /// <param name="state"></param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split<T>(this in ReadOnlySpan<char> value, char splitter, StringSplitOptions options, ReadOnlySpanAction<char, T> splitCb, T state) + { + //Alloc a span for char + ReadOnlySpan<char> cs = MemoryMarshal.CreateReadOnlySpan(ref splitter, 1); + //Call the split function on the span + Split(in value, cs, options, splitCb, state); + } + /// <summary> + /// Split a <see cref="ReadOnlySpan{T}"/> based on split value and pass it to the split delegate handler + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The sequence to split the string on</param> + /// <param name="options">String split options</param> + /// <param name="splitCb">The action to invoke when a split segment has been found</param> + /// <exception cref="ArgumentNullException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split(this in ReadOnlySpan<char> value, ReadOnlySpan<char> splitter, StringSplitOptions options, StatelessSpanAction splitCb) + { + //Create a SpanSplitDelegate with the non-typed delegate as the state argument + static void ssplitcb(ReadOnlySpan<char> param, StatelessSpanAction callback) => callback(param); + //Call split with the new callback delegate + Split(in value, splitter, options, ssplitcb, splitCb); + } + /// <summary> + /// Split a <see cref="ReadOnlySpan{T}"/> based on split value and pass it to the split delegate handler + /// </summary> + /// <param name="value"></param> + /// <param name="splitter">The character to split the string on</param> + /// <param name="options">String split options</param> + /// <param name="splitCb">The action to invoke when a split segment has been found</param> + /// <exception cref="ArgumentNullException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Split(this in ReadOnlySpan<char> value, char splitter, StringSplitOptions options, StatelessSpanAction splitCb) + { + //Create a SpanSplitDelegate with the non-typed delegate as the state argument + static void ssplitcb(ReadOnlySpan<char> param, StatelessSpanAction callback) => callback(param); + //Call split with the new callback delegate + Split(in value, splitter, options, ssplitcb, splitCb); + } + + /// <summary> + /// Gets the index of the end of the found sequence + /// </summary> + /// <param name="data"></param> + /// <param name="search">Sequence to search for within the current sequence</param> + /// <returns>the index of the end of the sequenc</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int EndOf(this in ReadOnlySpan<char> data, ReadOnlySpan<char> search) + { + int index = data.IndexOf(search); + return index > -1 ? index + search.Length : -1; + } + /// <summary> + /// Gets the index of the end of the found character + /// </summary> + /// <param name="data"></param> + /// <param name="search">Character to search for within the current sequence</param> + /// <returns>the index of the end of the sequence</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int EndOf(this in ReadOnlySpan<char> data, char search) + { + int index = data.IndexOf(search); + return index > -1 ? index + 1 : -1; + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int IndexOf(this in Memory<byte> data, byte search) => data.Span.IndexOf(search); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int IndexOf(this in Memory<byte> data, ReadOnlySpan<byte> search) => data.Span.IndexOf(search); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int IndexOf(this in Memory<byte> data, ReadOnlyMemory<byte> search) => IndexOf(data, search.Span); + + /// <summary> + /// Slices the current span from the begining of the segment to the first occurrance of the specified character. + /// If the character is not found, the entire segment is returned + /// </summary> + /// <param name="data"></param> + /// <param name="search">The delimiting character</param> + /// <returns>The segment of data before the search character, or the entire segment if not found</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan<char> SliceBeforeParam(this in ReadOnlySpan<char> data, char search) + { + //Find the index of the specified data + int index = data.IndexOf(search); + //Return the slice of data before the index, or an empty span if it was not found + return index > -1 ? data[..index] : data; + } + /// <summary> + /// Slices the current span from the begining of the segment to the first occurrance of the specified character sequence. + /// If the character sequence is not found, the entire segment is returned + /// </summary> + /// <param name="data"></param> + /// <param name="search">The delimiting character sequence</param> + /// <returns>The segment of data before the search character, or the entire <paramref name="data"/> if the seach sequence is not found</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan<char> SliceBeforeParam(this in ReadOnlySpan<char> data, ReadOnlySpan<char> search) + { + //Find the index of the specified data + int index = data.IndexOf(search); + //Return the slice of data before the index, or an empty span if it was not found + return index > -1 ? data[..index] : data; + } + /// <summary> + /// Gets the remaining segment of data after the specified search character or <see cref="ReadOnlySpan{T}.Empty"/> + /// if the search character is not found within the current segment + /// </summary> + /// <param name="data"></param> + /// <param name="search">The character to search for within the segment</param> + /// <returns>The segment of data after the search character or <see cref="ReadOnlySpan{T}.Empty"/> if not found</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan<char> SliceAfterParam(this in ReadOnlySpan<char> data, char search) + { + //Find the index of the specified data + int index = EndOf(in data, search); + //Return the slice of data after the index, or an empty span if it was not found + return index > -1 ? data[index..] : ReadOnlySpan<char>.Empty; + } + /// <summary> + /// Gets the remaining segment of data after the specified search sequence or <see cref="ReadOnlySpan{T}.Empty"/> + /// if the search sequence is not found within the current segment + /// </summary> + /// <param name="data"></param> + /// <param name="search">The sequence to search for within the segment</param> + /// <returns>The segment of data after the search sequence or <see cref="ReadOnlySpan{T}.Empty"/> if not found</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan<char> SliceAfterParam(this in ReadOnlySpan<char> data, ReadOnlySpan<char> search) + { + //Find the index of the specified data + int index = EndOf(data, search); + //Return the slice of data after the index, or an empty span if it was not found + return index > -1 ? data[index..] : ReadOnlySpan<char>.Empty; + } + /// <summary> + /// Trims any leading or trailing <c>'\r'|'\n'|' '</c>(whitespace) characters from the segment + /// </summary> + /// <returns>The trimmed segment</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan<char> TrimCRLF(this in ReadOnlySpan<char> data) + { + int start = 0, end = data.Length; + //trim leading \r\n chars + while(start < end) + { + char t = data[start]; + //If character \r or \n slice it off + if (t != '\r' && t != '\n' && t != ' ') { + break; + } + //Shift + start++; + } + //remove trailing crlf characters + while (end > start) + { + char t = data[end - 1]; + //If character \r or \n slice it off + if (t != '\r' && t != '\n' && t != ' ') { + break; + } + end--; + } + return data[start..end]; + } + + /// <summary> + /// Replaces a character sequence within the buffer + /// </summary> + /// <param name="buffer">The character buffer to process</param> + /// <param name="search">The sequence to search for</param> + /// <param name="replace">The sequence to write in the place of the search parameter</param> + /// <exception cref="OutOfMemoryException"></exception> + public static int Replace(this ref Span<char> buffer, ReadOnlySpan<char> search, ReadOnlySpan<char> replace) + { + ForwardOnlyWriter<char> writer = new (buffer); + writer.Replace(search, replace); + return writer.Written; + } + + /// <summary> + /// Replaces a character sequence within the writer + /// </summary> + /// <param name="writer"></param> + /// <param name="search">The sequence to search for</param> + /// <param name="replace">The sequence to write in the place of the search parameter</param> + /// <exception cref="OutOfMemoryException"></exception> + public static void Replace(this ref ForwardOnlyWriter<char> writer, ReadOnlySpan<char> search, ReadOnlySpan<char> replace) + { + Span<char> buffer = writer.AsSpan(); + //If the search and replacment parameters are the same length + if (search.Length == replace.Length) + { + buffer.ReplaceInPlace(search, replace); + return; + } + //Search and replace are not the same length + int searchLen = search.Length, start = buffer.IndexOf(search); + if(start == -1) + { + return; + } + //Replacment might be empty + writer.Reset(); + do + { + //Append the data before the split character + writer.Append(buffer[..start]); + //Append the replacment + writer.Append(replace); + //Shift buffer to the end of the + buffer = buffer[(start + searchLen)..]; + //search for next index + start = buffer.IndexOf(search); + } while (start > -1); + //Write remaining data + writer.Append(replace); + } + /// <summary> + /// Replaces very ocurrance of character sequence within a buffer with another sequence of the same length + /// </summary> + /// <param name="buffer"></param> + /// <param name="search">The sequence to search for</param> + /// <param name="replace">The sequence to replace the found sequence with</param> + /// <exception cref="ArgumentException"></exception> + public static void ReplaceInPlace(this Span<char> buffer, ReadOnlySpan<char> search, ReadOnlySpan<char> replace) + { + if(search.Length != replace.Length) + { + throw new ArgumentException("Search parameter and replacment parameter must be the same length"); + } + int start = buffer.IndexOf(search); + while(start > -1) + { + //Shift the buffer to the begining of the search parameter + buffer = buffer[start..]; + //Overwite the search parameter + replace.CopyTo(buffer); + //Search for next index of the search character + start = buffer.IndexOf(search); + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/ThreadingExtensions.cs b/Utils/src/Extensions/ThreadingExtensions.cs new file mode 100644 index 0000000..cc9fab9 --- /dev/null +++ b/Utils/src/Extensions/ThreadingExtensions.cs @@ -0,0 +1,226 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ThreadingExtensions.cs +* +* ThreadingExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Extensions +{ + + /// <summary> + /// Provides extension methods to common threading and TPL library operations + /// </summary> + public static class ThreadingExtensions + { + /// <summary> + /// Allows an <see cref="OpenResourceHandle{TResource}"/> to execute within a scope limited context + /// </summary> + /// <typeparam name="TResource">The resource type</typeparam> + /// <param name="rh"></param> + /// <param name="safeCallback">The function body that will execute with controlled access to the resource</param> + public static void EnterSafeContext<TResource>(this OpenResourceHandle<TResource> rh, Action<TResource> safeCallback) + { + using (rh) + { + safeCallback(rh.Resource); + } + } + + /// <summary> + /// Asynchronously waits to enter the <see cref="SemaphoreSlim"/> while observing a <see cref="CancellationToken"/> + /// and getting a releaser handle + /// </summary> + /// <param name="semaphore"></param> + /// <param name="cancellationToken">A token to cancel the operation</param> + /// <returns>A releaser handle that may be disposed to release the semaphore</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="OperationCanceledException"></exception> + public static async Task<SemSlimReleaser> GetReleaserAsync(this SemaphoreSlim semaphore, CancellationToken cancellationToken = default) + { + await semaphore.WaitAsync(cancellationToken); + return new SemSlimReleaser(semaphore); + } + /// <summary> + /// Asynchronously waits to enter the <see cref="SemaphoreSlim"/> using a 32-bit signed integer to measure the time intervale + /// and getting a releaser handle + /// </summary> + /// <param name="semaphore"></param> + /// <param name="timeout">A the maximum amount of time in milliseconds to wait to enter the semaphore</param> + /// <returns>A releaser handle that may be disposed to release the semaphore</returns> + /// <exception cref="TimeoutException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static async Task<SemSlimReleaser> GetReleaserAsync(this SemaphoreSlim semaphore, int timeout) + { + if (await semaphore.WaitAsync(timeout)) + { + return new SemSlimReleaser(semaphore); + } + throw new TimeoutException("Failed to enter the semaphore before the specified timeout period"); + } + + /// <summary> + /// Blocks the current thread until it can enter the <see cref="SemaphoreSlim"/> + /// </summary> + /// <param name="semaphore"></param> + /// <returns>A releaser handler that releases the semaphore when disposed</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static SemSlimReleaser GetReleaser(this SemaphoreSlim semaphore) + { + semaphore.Wait(); + return new SemSlimReleaser(semaphore); + } + /// <summary> + /// Blocks the current thread until it can enter the <see cref="SemaphoreSlim"/> + /// </summary> + /// <param name="semaphore"></param> + /// <param name="timeout">A the maximum amount of time in milliseconds to wait to enter the semaphore</param> + /// <returns>A releaser handler that releases the semaphore when disposed</returns> + /// <exception cref="TimeoutException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static SemSlimReleaser GetReleaser(this SemaphoreSlim semaphore, int timeout) + { + if (semaphore.Wait(timeout)) + { + return new SemSlimReleaser(semaphore); + } + throw new TimeoutException("Failed to enter the semaphore before the specified timeout period"); + } + + /// <summary> + /// Blocks the current thread until it can enter the <see cref="Mutex"/> + /// </summary> + /// <param name="mutex"></param> + /// <returns>A releaser handler that releases the semaphore when disposed</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="AbandonedMutexException"></exception> + public static MutexReleaser Enter(this Mutex mutex) + { + mutex.WaitOne(); + return new MutexReleaser(mutex); + } + /// <summary> + /// Blocks the current thread until it can enter the <see cref="SemaphoreSlim"/> + /// </summary> + /// <param name="mutex"></param> + /// <param name="timeout">A the maximum amount of time in milliseconds to wait to enter the semaphore</param> + /// <returns>A releaser handler that releases the semaphore when disposed</returns> + /// <exception cref="TimeoutException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static MutexReleaser Enter(this Mutex mutex, int timeout) + { + if (mutex.WaitOne(timeout)) + { + return new MutexReleaser(mutex); + } + throw new TimeoutException("Failed to enter the semaphore before the specified timeout period"); + } + + private static readonly Task<bool> TrueCompleted = Task.FromResult(true); + private static readonly Task<bool> FalseCompleted = Task.FromResult(false); + + /// <summary> + /// Asynchronously waits for a the <see cref="WaitHandle"/> to receive a signal. This method spins until + /// a thread yield will occur, then asynchronously yields. + /// </summary> + /// <param name="handle"></param> + /// <param name="timeoutMs">The timeout interval in milliseconds</param> + /// <returns> + /// A task that compeletes when the wait handle receives a signal or times-out, + /// the result of the awaited task will be <c>true</c> if the signal is received, or + /// <c>false</c> if the timeout interval expires + /// </returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static Task<bool> WaitAsync(this WaitHandle handle, int timeoutMs = Timeout.Infinite) + { + _ = handle ?? throw new ArgumentNullException(nameof(handle)); + //test non-blocking handle state + if (handle.WaitOne(0)) + { + return TrueCompleted; + } + //When timeout is 0, wh will block, return false + else if(timeoutMs == 0) + { + return FalseCompleted; + } + //Init short lived spinwait + SpinWait sw = new(); + //Spin until yield occurs + while (!sw.NextSpinWillYield) + { + sw.SpinOnce(); + //Check handle state + if (handle.WaitOne(0)) + { + return TrueCompleted; + } + } + //Completion source used to signal the awaiter when the wait handle is signaled + TaskCompletionSource<bool> completion = new(TaskCreationOptions.None); + //Register wait on threadpool to complete the task source + RegisteredWaitHandle registration = ThreadPool.RegisterWaitForSingleObject(handle, TaskCompletionCallback, completion, timeoutMs, true); + //Register continuation to cleanup + _ = completion.Task.ContinueWith(CleanupContinuation, registration, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default) + .ConfigureAwait(false); + return completion.Task; + } + + private static void CleanupContinuation(Task<bool> task, object? taskCompletion) + { + RegisteredWaitHandle registration = (taskCompletion as RegisteredWaitHandle)!; + registration.Unregister(null); + task.Dispose(); + } + private static void TaskCompletionCallback(object? tcsState, bool timedOut) + { + TaskCompletionSource<bool> completion = (tcsState as TaskCompletionSource<bool>)!; + //Set the result of the wait handle timeout + _ = completion.TrySetResult(!timedOut); + } + + + /// <summary> + /// Registers a callback method that will be called when the token has been cancelled. + /// This method waits indefinitely for the token to be cancelled. + /// </summary> + /// <param name="token"></param> + /// <param name="callback">The callback method to invoke when the token has been cancelled</param> + /// <returns>A task that may be unobserved, that completes when the token has been cancelled</returns> + public static Task RegisterUnobserved(this CancellationToken token, Action callback) + { + //Call callback when the wait handle is set + return token.WaitHandle.WaitAsync() + .ContinueWith(static (t, callback) => (callback as Action)!.Invoke(), + callback, + CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default + ); + } + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/TimerExtensions.cs b/Utils/src/Extensions/TimerExtensions.cs new file mode 100644 index 0000000..a980d63 --- /dev/null +++ b/Utils/src/Extensions/TimerExtensions.cs @@ -0,0 +1,66 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: TimerExtensions.cs +* +* TimerExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Extensions +{ + public static class TimerExtensions + { + /// <summary> + /// Attempts to stop the timer + /// </summary> + /// <returns>True if the timer was successfully modified, false otherwise</returns> + public static bool Stop(this Timer timer) => timer.Change(Timeout.Infinite, Timeout.Infinite); + + /// <summary> + /// Attempts to stop an active timer and prepare a <see cref="OpenHandle"/> configured to restore the state of the timer to the specified timespan + /// </summary> + /// <param name="timer"></param> + /// <param name="resumeTime"><see cref="TimeSpan"/> representing the amount of time the timer should wait before invoking the callback function</param> + /// <returns>A new <see cref="OpenHandle"/> if the timer was stopped successfully that will resume the timer when closed, null otherwise</returns> + public static OpenHandle Stop(this Timer timer, TimeSpan resumeTime) + { + return timer.Change(Timeout.Infinite, Timeout.Infinite) ? new CallbackOpenHandle(() => timer.Change(resumeTime, Timeout.InfiniteTimeSpan)) : null; + } + + /// <summary> + /// Attempts to reset and start a timer + /// </summary> + /// <param name="timer"></param> + /// <param name="wait"><see cref="TimeSpan"/> to wait before the timer event is fired</param> + /// <returns>True if the timer was successfully modified</returns> + public static bool Restart(this Timer timer, TimeSpan wait) => timer.Change(wait, Timeout.InfiniteTimeSpan); + + /// <summary> + /// Attempts to reset and start a timer + /// </summary> + /// <param name="timer"></param> + /// <param name="waitMilliseconds">Time in milliseconds to wait before the timer event is fired</param> + /// <returns>True if the timer was successfully modified</returns> + public static bool Restart(this Timer timer, int waitMilliseconds) => timer.Change(waitMilliseconds, Timeout.Infinite); + } +}
\ No newline at end of file diff --git a/Utils/src/Extensions/VnStringExtensions.cs b/Utils/src/Extensions/VnStringExtensions.cs new file mode 100644 index 0000000..285fc4f --- /dev/null +++ b/Utils/src/Extensions/VnStringExtensions.cs @@ -0,0 +1,418 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnStringExtensions.cs +* +* VnStringExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Linq; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; + +using VNLib.Utils.Memory; + +namespace VNLib.Utils.Extensions +{ + [SuppressMessage("Design", "CA1062:Validate arguments of public methods", Justification = "<Pending>")] + public static class VnStringExtensions + { + /// <summary> + /// Derermines if the character exists within the instance + /// </summary> + /// <param name="str"></param> + /// <param name="value">The value to find</param> + /// <returns>True if the character exists within the instance</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static bool Contains(this VnString str, char value) => str.AsSpan().Contains(value); + /// <summary> + /// Derermines if the sequence exists within the instance + /// </summary> + /// <param name="str"></param> + /// <param name="value">The sequence to find</param> + /// <param name="stringComparison"></param> + /// <returns>True if the character exists within the instance</returns> + /// <exception cref="ObjectDisposedException"></exception> + + public static bool Contains(this VnString str, ReadOnlySpan<char> value, StringComparison stringComparison) => str.AsSpan().Contains(value, stringComparison); + + /// <summary> + /// Searches for the first occurrance of the specified character within the current instance + /// </summary> + /// <param name="str"></param> + /// <param name="value">The character to search for within the instance</param> + /// <returns>The 0 based index of the occurance, -1 if the character was not found</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static int IndexOf(this VnString str, char value) => str.IsEmpty ? -1 : str.AsSpan().IndexOf(value); + /// <summary> + /// Searches for the first occurrance of the specified sequence within the current instance + /// </summary> + /// <param name="str"></param> + /// <param name="search">The sequence to search for</param> + /// <returns>The 0 based index of the occurance, -1 if the sequence was not found</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static int IndexOf(this VnString str, ReadOnlySpan<char> search) + { + //Using spans to avoid memory leaks... + ReadOnlySpan<char> self = str.AsSpan(); + return self.IndexOf(search); + } + /// <summary> + /// Searches for the first occurrance of the specified sequence within the current instance + /// </summary> + /// <param name="str"></param> + /// <param name="search">The sequence to search for</param> + /// <param name="comparison">The <see cref="StringComparison"/> type to use in searchr</param> + /// <returns>The 0 based index of the occurance, -1 if the sequence was not found</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static int IndexOf(this VnString str, ReadOnlySpan<char> search, StringComparison comparison) + { + //Using spans to avoid memory leaks... + ReadOnlySpan<char> self = str.AsSpan(); + return self.IndexOf(search, comparison); + } + /// <summary> + /// Searches for the 0 based index of the first occurance of the search parameter after the start index. + /// </summary> + /// <param name="str"></param> + /// <param name="search">The sequence of data to search for</param> + /// <param name="start">The lower boundry of the search area</param> + /// <returns>The absolute index of the first occurrance within the instance, -1 if the sequency was not found in the windowed segment</returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public static int IndexOf(this VnString str, ReadOnlySpan<char> search, int start) + { + if (start < 0) + { + throw new ArgumentOutOfRangeException(nameof(start), "Start cannot be less than 0"); + } + //Get shifted window + ReadOnlySpan<char> self = str.AsSpan()[start..]; + //Check indexof + int index = self.IndexOf(search); + return index > -1 ? index + start : -1; + } + + /// <summary> + /// Returns the realtive index after the specified sequence within the <see cref="VnString"/> instance + /// </summary> + /// <param name="str"></param> + /// <param name="search">The sequence to search for</param> + /// <returns>The index after the found sequence within the string, -1 if the sequence was not found within the instance</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static int EndOf(this VnString str, ReadOnlySpan<char> search) + { + //Try to get the index of the data + int index = IndexOf(str, search); + //If the data was found, add the length to get the end of the string + return index > -1 ? index + search.Length : -1; + } + + /// <summary> + /// Allows for trimming whitespace characters in a realtive sequence from + /// within a <see cref="VnString"/> buffer defined by the start and end parameters + /// and returning the trimmed entry. + /// </summary> + /// <param name="data"></param> + /// <param name="start">The starting position within the sequence to trim</param> + /// <param name="end">The end of the sequence to trim</param> + /// <returns>The trimmed <see cref="VnString"/> instance as a child of the original entry</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static VnString AbsoluteTrim(this VnString data, int start, int end) + { + AbsoluteTrim(data, ref start, ref end); + return data[start..end]; + } + /// <summary> + /// Finds whitespace characters within the sequence defined between start and end parameters + /// and adjusts the specified window to "trim" whitespace + /// </summary> + /// <param name="data"></param> + /// <param name="start">The starting position within the sequence to trim</param> + /// <param name="end">The end of the sequence to trim</param> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static void AbsoluteTrim(this VnString data, ref int start, ref int end) + { + ReadOnlySpan<char> trimmed = data.AsSpan(); + //trim leading whitespace + while (start < end) + { + //If whitespace character shift start up + if (trimmed[start] != ' ') + { + break; + } + //Shift + start++; + } + //remove trailing whitespace characters + while (end > start) + { + //If whiterspace character shift end param down + if (trimmed[end - 1] != ' ') + { + break; + } + end--; + } + } + /// <summary> + /// Allows for trimming whitespace characters in a realtive sequence from + /// within a <see cref="VnString"/> buffer and returning the trimmed entry. + /// </summary> + /// <param name="data"></param> + /// <param name="start">The starting position within the sequence to trim</param> + /// <returns>The trimmed <see cref="VnString"/> instance as a child of the original entry</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="IndexOutOfRangeException"></exception> + public static VnString AbsoluteTrim(this VnString data, int start) => AbsoluteTrim(data, start, data.Length); + /// <summary> + /// Trims leading or trailing whitespace characters and returns a new child instance + /// without leading or trailing whitespace + /// </summary> + /// <returns>A child <see cref="VnString"/> of the current instance without leading or trailing whitespaced</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static VnString RelativeTirm(this VnString data) => AbsoluteTrim(data, 0); + + /// <summary> + /// Allows for enumeration of segments of data within the specified <see cref="VnString"/> instance that are + /// split by the search parameter + /// </summary> + /// <param name="data"></param> + /// <param name="search">The sequence of data to delimit segments</param> + /// <param name="options">The options used to split the string instances</param> + /// <returns>An iterator to enumerate the split segments</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static IEnumerable<VnString> Split(this VnString data, ReadOnlyMemory<char> search, StringSplitOptions options = StringSplitOptions.None) + { + int lowerBound = 0; + //Make sure the length of the search param is not 0 + if(search.IsEmpty) + { + //Return the entire string + yield return data; + } + //No string options + else if (options == 0) + { + do + { + //Capture the first = and store argument + value + int splitIndex = data.IndexOf(search.Span, lowerBound); + //If no split index is found, then return remaining data + if (splitIndex == -1) + { + break; + } + yield return data[lowerBound..splitIndex]; + //Shift the lower window to the end of the last string + lowerBound = splitIndex + search.Length; + } while (true); + //Return remaining data + yield return data[lowerBound..]; + } + //Trim but do not remove empties + else if ((options & StringSplitOptions.RemoveEmptyEntries) == 0) + { + do + { + //Capture the first = and store argument + value + int splitIndex = data.IndexOf(search.Span, lowerBound); + //If no split index is found, then return remaining data + if (splitIndex == -1) + { + break; + } + //trim and return + yield return data.AbsoluteTrim(lowerBound, splitIndex); + //Shift the lower window to the end of the last string + lowerBound = splitIndex + search.Length; + } while (true); + //Return remaining data + yield return data.AbsoluteTrim(lowerBound); + } + //Remove empty entires but do not trim them + else if ((options & StringSplitOptions.TrimEntries) == 0) + { + do + { + //Capture the first = and store argument + value + int splitIndex = data.IndexOf(search.Span, lowerBound); + //If no split index is found, then return remaining data + if (splitIndex == -1) + { + break; + } + //If the split index is the next sequence, then the result is empty, so exclude it + else if(splitIndex > 0) + { + yield return data[lowerBound..splitIndex]; + } + //Shift the lower window to the end of the last string + lowerBound = splitIndex + search.Length; + } while (true); + //Return remaining data if available + if (lowerBound < data.Length) + { + yield return data[lowerBound..]; + } + } + //Must mean remove and trim + else + { + //Get stack varables to pass to trim function + int trimStart, trimEnd; + do + { + //Capture the first = and store argument + value + int splitIndex = data.IndexOf(search.Span, lowerBound); + //If no split index is found, then return remaining data + if (splitIndex == -1) + { + break; + } + //Get stack varables to pass to trim function + trimStart = lowerBound; + trimEnd = splitIndex; //End of the segment is the relative split index + the lower bound of the window + //Trim whitespace chars + data.AbsoluteTrim(ref trimStart, ref trimEnd); + //See if the string has data + if((trimEnd - trimStart) > 0) + { + yield return data[trimStart..trimEnd]; + } + //Shift the lower window to the end of the last string + lowerBound = splitIndex + search.Length; + } while (true); + //Trim remaining + trimStart = lowerBound; + trimEnd = data.Length; + data.AbsoluteTrim(ref trimStart, ref trimEnd); + //If the remaining string is not empty return it + if ((trimEnd - trimStart) > 0) + { + yield return data[trimStart..trimEnd]; + } + } + } + + /// <summary> + /// Trims any leading or trailing <c>'\r'|'\n'|' '</c>(whitespace) characters from the segment + /// </summary> + /// <returns>The trimmed segment</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static VnString TrimCRLF(this VnString data) + { + ReadOnlySpan<char> trimmed = data.AsSpan(); + int start = 0, end = trimmed.Length; + //trim leading \r\n chars + while (start < end) + { + char t = trimmed[start]; + //If character \r or \n slice it off + if (t != '\r' && t != '\n' && t != ' ') { + break; + } + //Shift + start++; + } + //remove trailing crlf characters + while (end > start) + { + char t = trimmed[end - 1]; + //If character \r or \n slice it off + if (t != '\r' && t != '\n' && t != ' ') { + break; + } + end--; + } + return data[start..end]; + } + + /// <summary> + /// Unoptimized character enumerator. You should use <see cref="VnString.AsSpan"/> to enumerate the unerlying data. + /// </summary> + /// <returns>The next character in the sequence</returns> + /// <exception cref="ObjectDisposedException"></exception> + public static IEnumerator<char> GetEnumerator(this VnString data) + { + int index = 0; + while (index < data.Length) + { + yield return data[index++]; + } + } + /// <summary> + /// Converts the current handle to a <see cref="VnString"/>, a zero-alloc immutable wrapper + /// for a memory handle + /// </summary> + /// <param name="handle"></param> + /// <param name="length">The number of characters from the handle to reference (length of the string)</param> + /// <returns>The new <see cref="VnString"/> wrapper</returns> + /// <exception cref="OverflowException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static VnString ToVnString(this MemoryHandle<char> handle, int length) + { + if(handle.Length > int.MaxValue) + { + throw new OverflowException("The handle is larger than 2GB in size"); + } + return VnString.ConsumeHandle(handle, 0, length); + } + /// <summary> + /// Converts the current handle to a <see cref="VnString"/>, a zero-alloc immutable wrapper + /// for a memory handle + /// </summary> + /// <param name="handle"></param> + /// <returns>The new <see cref="VnString"/> wrapper</returns> + /// <exception cref="OverflowException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static VnString ToVnString(this MemoryHandle<char> handle) + { + return VnString.ConsumeHandle(handle, 0, handle.IntLength); + } + /// <summary> + /// Converts the current handle to a <see cref="VnString"/>, a zero-alloc immutable wrapper + /// for a memory handle + /// </summary> + /// <param name="handle"></param> + /// <param name="offset">The offset in characters that represents the begining of the string</param> + /// <param name="length">The number of characters from the handle to reference (length of the string)</param> + /// <returns>The new <see cref="VnString"/> wrapper</returns> + /// <exception cref="OverflowException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static VnString ToVnString(this MemoryHandle<char> handle, +#if TARGET_64_BIT + ulong offset, +#else + int offset, +#endif + int length) + { + if (handle.Length > int.MaxValue) + { + throw new OverflowException("The handle is larger than 2GB in size"); + } + return VnString.ConsumeHandle(handle, offset, length); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IIndexable.cs b/Utils/src/IIndexable.cs new file mode 100644 index 0000000..129d703 --- /dev/null +++ b/Utils/src/IIndexable.cs @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IIndexable.cs +* +* IIndexable.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils +{ + /// <summary> + /// Provides an interface that provides an indexer + /// </summary> + /// <typeparam name="TKey">The lookup Key</typeparam> + /// <typeparam name="TValue">The lookup value</typeparam> + public interface IIndexable<TKey, TValue> + { + /// <summary> + /// Gets or sets the value at the specified index in the collection + /// </summary> + /// <param name="key">The key to lookup the value at</param> + /// <returns>The value at the specified key</returns> + TValue this[TKey key] { get; set;} + } +} diff --git a/Utils/src/IO/ArrayPoolStreamBuffer.cs b/Utils/src/IO/ArrayPoolStreamBuffer.cs new file mode 100644 index 0000000..df366e3 --- /dev/null +++ b/Utils/src/IO/ArrayPoolStreamBuffer.cs @@ -0,0 +1,70 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ArrayPoolStreamBuffer.cs +* +* ArrayPoolStreamBuffer.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; + +namespace VNLib.Utils.IO +{ + internal class ArrayPoolStreamBuffer<T> : ISlindingWindowBuffer<T> + { + private readonly ArrayPool<T> _pool; + private T[] _buffer; + + public ArrayPoolStreamBuffer(ArrayPool<T> pool, int bufferSize) + { + _pool = pool; + _buffer = _pool.Rent(bufferSize); + } + + public int WindowStartPos { get; set; } + public int WindowEndPos { get; set; } + + public Memory<T> Buffer => _buffer.AsMemory(); + + public void Advance(int count) + { + WindowEndPos += count; + } + + public void AdvanceStart(int count) + { + WindowStartPos += count; + } + + public void Close() + { + //Return buffer to pool + _pool.Return(_buffer); + _buffer = null; + } + + public void Reset() + { + //Reset window positions + WindowStartPos = 0; + WindowEndPos = 0; + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/BackingStream.cs b/Utils/src/IO/BackingStream.cs new file mode 100644 index 0000000..cb56b09 --- /dev/null +++ b/Utils/src/IO/BackingStream.cs @@ -0,0 +1,181 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: BackingStream.cs +* +* BackingStream.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Provides basic stream support sync/async stream operations to a + /// backing stream with virtual event methods. Provides a pass-through + /// as best as possbile. + /// </summary> + public abstract class BackingStream<T> : Stream where T: Stream + { + /// <summary> + /// The backing/underlying stream operations are being performed on + /// </summary> + protected T BaseStream { get; set; } + /// <summary> + /// A value that will cause all calls to write to throw <see cref="NotSupportedException"/> + /// </summary> + protected bool ForceReadOnly { get; set; } + ///<inheritdoc/> + public override bool CanRead => BaseStream.CanRead; + ///<inheritdoc/> + public override bool CanSeek => BaseStream.CanSeek; + ///<inheritdoc/> + public override bool CanWrite => BaseStream.CanWrite && !ForceReadOnly; + ///<inheritdoc/> + public override long Length => BaseStream.Length; + ///<inheritdoc/> + public override int WriteTimeout { get => BaseStream.WriteTimeout; set => BaseStream.WriteTimeout = value; } + ///<inheritdoc/> + public override int ReadTimeout { get => BaseStream.ReadTimeout; set => BaseStream.ReadTimeout = value; } + ///<inheritdoc/> + public override long Position { get => BaseStream.Position; set => BaseStream.Position = value; } + ///<inheritdoc/> + public override void Flush() + { + BaseStream.Flush(); + OnFlush(); + } + ///<inheritdoc/> + public override int Read(byte[] buffer, int offset, int count) => BaseStream.Read(buffer, offset, count); + ///<inheritdoc/> + public override int Read(Span<byte> buffer) => BaseStream.Read(buffer); + ///<inheritdoc/> + public override long Seek(long offset, SeekOrigin origin) => BaseStream.Seek(offset, origin); + ///<inheritdoc/> + public override void SetLength(long value) => BaseStream.SetLength(value); + ///<inheritdoc/> + public override void Write(byte[] buffer, int offset, int count) + { + if (ForceReadOnly) + { + throw new NotSupportedException("Stream is set to readonly mode"); + } + BaseStream.Write(buffer, offset, count); + //Call onwrite function + OnWrite(count); + } + ///<inheritdoc/> + public override void Write(ReadOnlySpan<byte> buffer) + { + if (ForceReadOnly) + { + throw new NotSupportedException("Stream is set to readonly mode"); + } + BaseStream.Write(buffer); + //Call onwrite function + OnWrite(buffer.Length); + } + ///<inheritdoc/> + public override void Close() + { + BaseStream.Close(); + //Call on close function + OnClose(); + } + + /// <summary> + /// Raised directly after the base stream is closed, when a call to close is made + /// </summary> + protected virtual void OnClose() { } + /// <summary> + /// Raised directly after the base stream is flushed, when a call to flush is made + /// </summary> + protected virtual void OnFlush() { } + /// <summary> + /// Raised directly after a successfull write operation. + /// </summary> + /// <param name="count">The number of bytes written to the stream</param> + protected virtual void OnWrite(int count) { } + + ///<inheritdoc/> + public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + return BaseStream.ReadAsync(buffer, offset, count, cancellationToken); + } + ///<inheritdoc/> + public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default) + { + return BaseStream.ReadAsync(buffer, cancellationToken); + } + ///<inheritdoc/> + public override void CopyTo(Stream destination, int bufferSize) => BaseStream.CopyTo(destination, bufferSize); + ///<inheritdoc/> + public override Task CopyToAsync(Stream destination, int bufferSize, CancellationToken cancellationToken) + { + return BaseStream.CopyToAsync(destination, bufferSize, cancellationToken); + } + ///<inheritdoc/> + public override async Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + if (ForceReadOnly) + { + throw new NotSupportedException("Stream is set to readonly mode"); + } + //We want to maintain pass through as much as possible, so supress warning +#pragma warning disable CA1835 // Prefer the 'Memory'-based overloads for 'ReadAsync' and 'WriteAsync' + await BaseStream.WriteAsync(buffer, offset, count, cancellationToken); +#pragma warning restore CA1835 // Prefer the 'Memory'-based overloads for 'ReadAsync' and 'WriteAsync' + + //Call on-write and pass the number of bytes written + OnWrite(count); + } + ///<inheritdoc/> + public override async ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default) + { + if (ForceReadOnly) + { + throw new NotSupportedException("Stream is set to readonly mode"); + } + await BaseStream.WriteAsync(buffer, cancellationToken); + //Call on-write and pass the length + OnWrite(buffer.Length); + } + ///<inheritdoc/> + public override async Task FlushAsync(CancellationToken cancellationToken) + { + await BaseStream.FlushAsync(cancellationToken); + //Call onflush + OnFlush(); + } + + ///<inheritdoc/> + public override async ValueTask DisposeAsync() + { + //Dispose the base stream and await it + await BaseStream.DisposeAsync(); + //Call onclose + OnClose(); + //Suppress finalize + GC.SuppressFinalize(this); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/FileOperations.cs b/Utils/src/IO/FileOperations.cs new file mode 100644 index 0000000..e040da4 --- /dev/null +++ b/Utils/src/IO/FileOperations.cs @@ -0,0 +1,105 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: FileOperations.cs +* +* FileOperations.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Runtime.InteropServices; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Contains cross-platform optimized filesystem operations. + /// </summary> + public static class FileOperations + { + public const int INVALID_FILE_ATTRIBUTES = -1; + + [DllImport("Shlwapi", SetLastError = true, CharSet = CharSet.Auto)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + [return:MarshalAs(UnmanagedType.Bool)] + private static unsafe extern bool PathFileExists(char* path); + [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Auto)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + [return:MarshalAs(UnmanagedType.I4)] + private static unsafe extern int GetFileAttributes(char* path); + + static readonly bool IsWindows = OperatingSystem.IsWindows(); + /// <summary> + /// Determines if a file exists. If application is current running in the Windows operating system, Shlwapi.PathFileExists is invoked, + /// otherwise <see cref="File.Exists(string?)"/> is invoked + /// </summary> + /// <param name="filePath">the path to the file</param> + /// <returns>True if the file can be opened, false otherwise</returns> + public static bool FileExists(string filePath) + { + //If windows is detected, use the unmanged function + if (!IsWindows) + { + return File.Exists(filePath); + } + unsafe + { + //Get a char pointer to the file path + fixed (char* path = filePath) + { + //Invoke the winap file function + return PathFileExists(path); + } + } + } + + /// <summary> + /// If Windows is detected at load time, gets the attributes for the specified file. + /// </summary> + /// <param name="filePath">The path to the existing file</param> + /// <returns>The attributes of the file </returns> + /// <exception cref="PathTooLongException"></exception> + /// <exception cref="FileNotFoundException"></exception> + /// <exception cref="UnauthorizedAccessException"></exception> + public static FileAttributes GetAttributes(string filePath) + { + //If windows is detected, use the unmanged function + if (!IsWindows) + { + return File.GetAttributes(filePath); + } + unsafe + { + //Get a char pointer to the file path + fixed (char* path = filePath) + { + //Invoke the winap file function and cast the returned int value to file attributes + int attr = GetFileAttributes(path); + //Check for error + if (attr == INVALID_FILE_ATTRIBUTES) + { + throw new FileNotFoundException("The requested file was not found", filePath); + } + //Cast to file attributes and return + return (FileAttributes)attr; + } + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/IDataAccumulator.cs b/Utils/src/IO/IDataAccumulator.cs new file mode 100644 index 0000000..5129a55 --- /dev/null +++ b/Utils/src/IO/IDataAccumulator.cs @@ -0,0 +1,64 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IDataAccumulator.cs +* +* IDataAccumulator.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// A data structure that represents a sliding window over a buffer + /// for resetable forward-only reading or writing + /// </summary> + /// <typeparam name="T">The accumuation data type</typeparam> + public interface IDataAccumulator<T> + { + /// <summary> + /// Gets the number of available items within the buffer + /// </summary> + int AccumulatedSize { get; } + /// <summary> + /// The number of elements remaining in the buffer + /// </summary> + int RemainingSize { get; } + /// <summary> + /// The remaining space in the internal buffer as a contiguous segment + /// </summary> + Span<T> Remaining { get; } + /// <summary> + /// The buffer window over the accumulated data + /// </summary> + Span<T> Accumulated { get; } + + /// <summary> + /// Advances the accumulator buffer window by the specified amount + /// </summary> + /// <param name="count">The number of elements accumulated</param> + void Advance(int count); + + /// <summary> + /// Resets the internal state of the accumulator + /// </summary> + void Reset(); + } +}
\ No newline at end of file diff --git a/Utils/src/IO/ISlindingWindowBuffer.cs b/Utils/src/IO/ISlindingWindowBuffer.cs new file mode 100644 index 0000000..ff4e142 --- /dev/null +++ b/Utils/src/IO/ISlindingWindowBuffer.cs @@ -0,0 +1,91 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ISlindingWindowBuffer.cs +* +* ISlindingWindowBuffer.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Represents a sliding window buffer for reading/wiriting data + /// </summary> + /// <typeparam name="T"></typeparam> + public interface ISlindingWindowBuffer<T> : IDataAccumulator<T> + { + /// <summary> + /// The number of elements remaining in the buffer + /// </summary> + int IDataAccumulator<T>.RemainingSize => Buffer.Length - WindowEndPos; + /// <summary> + /// The remaining space in the internal buffer as a contiguous segment + /// </summary> + Span<T> IDataAccumulator<T>.Remaining => RemainingBuffer.Span; + /// <summary> + /// The buffer window over the accumulated data + /// </summary> + Span<T> IDataAccumulator<T>.Accumulated => AccumulatedBuffer.Span; + /// <summary> + /// Gets the number of available items within the buffer + /// </summary> + int IDataAccumulator<T>.AccumulatedSize => WindowEndPos - WindowStartPos; + + /// <summary> + /// The starting positon of the available data within the buffer + /// </summary> + int WindowStartPos { get; } + /// <summary> + /// The ending position of the available data within the buffer + /// </summary> + int WindowEndPos { get; } + /// <summary> + /// Buffer memory wrapper + /// </summary> + Memory<T> Buffer { get; } + + /// <summary> + /// Releases resources used by the current instance + /// </summary> + void Close(); + /// <summary> + /// <para> + /// Advances the begining of the accumulated data window. + /// </para> + /// <para> + /// This method is used during reading to singal that data + /// has been read from the internal buffer and the + /// accumulator window can be shifted. + /// </para> + /// </summary> + /// <param name="count">The number of elements to shift by</param> + void AdvanceStart(int count); + + /// <summary> + /// Gets a window within the buffer of available buffered data + /// </summary> + Memory<T> AccumulatedBuffer => Buffer[WindowStartPos..WindowEndPos]; + /// <summary> + /// Gets the available buffer window to write data to + /// </summary> + Memory<T> RemainingBuffer => Buffer[WindowEndPos..]; + } +}
\ No newline at end of file diff --git a/Utils/src/IO/IVnTextReader.cs b/Utils/src/IO/IVnTextReader.cs new file mode 100644 index 0000000..625ba78 --- /dev/null +++ b/Utils/src/IO/IVnTextReader.cs @@ -0,0 +1,72 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IVnTextReader.cs +* +* IVnTextReader.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Represents a streaming text reader with internal buffers + /// </summary> + public interface IVnTextReader + { + /// <summary> + /// The base stream to read data from + /// </summary> + Stream BaseStream { get; } + /// <summary> + /// The character encoding used by the TextReader + /// </summary> + Encoding Encoding { get; } + /// <summary> + /// Number of available bytes of buffered data within the current buffer window + /// </summary> + int Available { get; } + /// <summary> + /// Gets or sets the line termination used to deliminate a line of data + /// </summary> + ReadOnlyMemory<byte> LineTermination { get; } + /// <summary> + /// The unread/available data within the internal buffer + /// </summary> + Span<byte> BufferedDataWindow { get; } + /// <summary> + /// Shifts the sliding buffer window by the specified number of bytes. + /// </summary> + /// <param name="count">The number of bytes read from the buffer</param> + void Advance(int count); + /// <summary> + /// Reads data from the stream into the remaining buffer space for processing + /// </summary> + void FillBuffer(); + /// <summary> + /// Compacts the available buffer space back to the begining of the buffer region + /// and determines if there is room for more data to be buffered + /// </summary> + /// <returns>The remaining buffer space if any</returns> + ERRNO CompactBufferWindow(); + } +}
\ No newline at end of file diff --git a/Utils/src/IO/InMemoryTemplate.cs b/Utils/src/IO/InMemoryTemplate.cs new file mode 100644 index 0000000..12f9092 --- /dev/null +++ b/Utils/src/IO/InMemoryTemplate.cs @@ -0,0 +1,196 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: InMemoryTemplate.cs +* +* InMemoryTemplate.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Represents a lazily loaded file stored in memory, with a change mointor + /// that reloads the template if the file was modified in the filesystem + /// </summary> + public abstract class InMemoryTemplate : VnDisposeable + { + protected ManualResetEventSlim TemplateLock; + private readonly FileSystemWatcher? Watcher; + private bool Modified; + private VnMemoryStream templateBuffer; + protected readonly FileInfo TemplateFile; + + /// <summary> + /// Gets the name of the template + /// </summary> + public abstract string TemplateName { get; } + + /// <summary> + /// Creates a new in-memory copy of a file that will detect changes and refresh + /// </summary> + /// <param name="listenForChanges">Should changes to the template file be moniored for changes, and reloaded as necessary</param> + /// <param name="path">The path of the file template</param> + protected InMemoryTemplate(string path, bool listenForChanges = true) + { + TemplateFile = new FileInfo(path); + TemplateLock = new(true); + //Make sure the file exists + if (!TemplateFile.Exists) + { + throw new FileNotFoundException("Template file does not exist"); + } + if (listenForChanges) + { + //Setup a watcher to reload the template when modified + Watcher = new FileSystemWatcher(TemplateFile.DirectoryName) + { + EnableRaisingEvents = true, + IncludeSubdirectories = false, + NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size + }; + Watcher.Changed += Watcher_Changed; + } + //Set modified flag to make sure the template is read on first use + this.Modified = true; + } + + private void Watcher_Changed(object sender, FileSystemEventArgs e) + { + //Make sure the event was raied for this template + if (!e.FullPath.Equals(TemplateFile.FullName, StringComparison.OrdinalIgnoreCase)) + { + return; + } + TemplateLock.Reset(); + try + { + //Set modified flag + Modified = true; + //Refresh the fileinfo object + TemplateFile.Refresh(); + //Invoke onmodifed function + OnModifed(); + } + finally + { + TemplateLock.Set(); + } + } + + /// <summary> + /// Gets a cached copy of the template data + /// </summary> + protected VnMemoryStream GetTemplateData() + { + //Make sure access is synchronized incase the file gets updated during access on another thread + TemplateLock.Wait(); + //Determine if the file has been modified and needs to be reloaded + if (Modified) + { + TemplateLock.Reset(); + try + { + //Read a new copy of the templte into mem + ReadFile(); + } + finally + { + TemplateLock.Set(); + } + } + //Return a copy of the memory stream + return templateBuffer.GetReadonlyShallowCopy(); + } + /// <summary> + /// Updates the internal copy of the file to its memory representation + /// </summary> + protected void ReadFile() + { + //Open the file stream + using FileStream fs = TemplateFile.OpenRead(); + //Dispose the old template buffer + templateBuffer?.Dispose(); + //Create a new stream for storing the cached copy + VnMemoryStream newBuf = new(); + try + { + fs.CopyTo(newBuf, null); + } + catch + { + newBuf.Dispose(); + throw; + } + //Create the readonly copy + templateBuffer = VnMemoryStream.CreateReadonly(newBuf); + //Clear the modified flag + Modified = false; + } + /// <summary> + /// Updates the internal copy of the file to its memory representation, asynchronously + /// </summary> + /// <param name="cancellationToken"></param> + /// <returns>A task that completes when the file has been copied into memory</returns> + protected async Task ReadFileAsync(CancellationToken cancellationToken = default) + { + //Open the file stream + await using FileStream fs = TemplateFile.OpenRead(); + //Dispose the old template buffer + templateBuffer?.Dispose(); + //Create a new stream for storing the cached copy + VnMemoryStream newBuf = new(); + try + { + //Copy async + await fs.CopyToAsync(newBuf, 8192, Memory.Memory.Shared, cancellationToken); + } + catch + { + newBuf.Dispose(); + throw; + } + //Create the readonly copy + templateBuffer = VnMemoryStream.CreateReadonly(newBuf); + //Clear the modified flag + Modified = false; + } + + /// <summary> + /// Invoked when the template file has been modifed. Note: This event is raised + /// while the <see cref="TemplateLock"/> is held. + /// </summary> + protected abstract void OnModifed(); + + ///<inheritdoc/> + protected override void Free() + { + //Dispose the watcher + Watcher?.Dispose(); + //free the stream + templateBuffer?.Dispose(); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/IsolatedStorageDirectory.cs b/Utils/src/IO/IsolatedStorageDirectory.cs new file mode 100644 index 0000000..65460ff --- /dev/null +++ b/Utils/src/IO/IsolatedStorageDirectory.cs @@ -0,0 +1,154 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IsolatedStorageDirectory.cs +* +* IsolatedStorageDirectory.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.IO.IsolatedStorage; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Represents an open directory within an <see cref="IsolatedStorageFile"/> store for which files can be created, opened, or deleted. + /// </summary> + public sealed class IsolatedStorageDirectory : IsolatedStorage + { + private readonly string DirectoryPath; + private readonly IsolatedStorageFile Storage; + /// <summary> + /// Creates a new <see cref="IsolatedStorageDirectory"/> within the specified file using the directory name. + /// </summary> + /// <param name="storage">A configured and open <see cref="IsolatedStorageFile"/></param> + /// <param name="dir">The directory name to open or create within the store</param> + public IsolatedStorageDirectory(IsolatedStorageFile storage, string dir) + { + this.Storage = storage; + this.DirectoryPath = dir; + //If the directory doesnt exist, create it + if (!this.Storage.DirectoryExists(dir)) + this.Storage.CreateDirectory(dir); + } + + private IsolatedStorageDirectory(IsolatedStorageDirectory parent, string dirName) + { + //Store ref to parent dir + Parent = parent; + //Referrence store + this.Storage = parent.Storage; + //Add the name of this dir to the end of the specified dir path + this.DirectoryPath = Path.Combine(parent.DirectoryPath, dirName); + } + + /// <summary> + /// Creates a file by its path name within the currnet directory + /// </summary> + /// <param name="fileName">The name of the file</param> + /// <returns>The open file</returns> + /// <exception cref="IsolatedStorageException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="DirectoryNotFoundException"></exception> + public IsolatedStorageFileStream CreateFile(string fileName) + { + return this.Storage.CreateFile(Path.Combine(DirectoryPath, fileName)); + } + /// <summary> + /// Removes a file from the current directory + /// </summary> + /// <param name="fileName">The path of the file to remove</param> + /// <exception cref="IsolatedStorageException"></exception> + public void DeleteFile(string fileName) + { + this.Storage.DeleteFile(Path.Combine(this.DirectoryPath, fileName)); + } + /// <summary> + /// Opens a file that exists within the current directory + /// </summary> + /// <param name="fileName">Name with extension of the file</param> + /// <param name="mode">File mode</param> + /// <param name="access">File access</param> + /// <returns>The open <see cref="IsolatedStorageFileStream"/> from the current directory</returns> + public IsolatedStorageFileStream OpenFile(string fileName, FileMode mode, FileAccess access) + { + return this.Storage.OpenFile(Path.Combine(DirectoryPath, fileName), mode, access); + } + /// <summary> + /// Opens a file that exists within the current directory + /// </summary> + /// <param name="fileName">Name with extension of the file</param> + /// <param name="mode">File mode</param> + /// <param name="access">File access</param> + /// <param name="share">The file shareing mode</param> + /// <returns>The open <see cref="IsolatedStorageFileStream"/> from the current directory</returns> + public IsolatedStorageFileStream OpenFile(string fileName, FileMode mode, FileAccess access, FileShare share) + { + return this.Storage.OpenFile(Path.Combine(DirectoryPath, fileName), mode, access, share); + } + + /// <summary> + /// Determiens if the specified file path refers to an existing file within the directory + /// </summary> + /// <param name="fileName">The name of the file to search for</param> + /// <returns>True if the file exists within the current directory</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="IsolatedStorageException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public bool FileExists(string fileName) + { + return this.Storage.FileExists(Path.Combine(this.DirectoryPath, fileName)); + } + + /// <summary> + /// Removes the directory and its contents from the store + /// </summary> + public override void Remove() + { + Storage.DeleteDirectory(this.DirectoryPath); + } + + public override long AvailableFreeSpace => Storage.AvailableFreeSpace; + public override long Quota => Storage.Quota; + public override long UsedSize => Storage.UsedSize; + public override bool IncreaseQuotaTo(long newQuotaSize) => Storage.IncreaseQuotaTo(newQuotaSize); + + /// <summary> + /// The parent <see cref="IsolatedStorageDirectory"/> this directory is a child within. null if there are no parent directories + /// above this dir + /// </summary> + + public IsolatedStorageDirectory? Parent { get; } +#nullable disable + + /// <summary> + /// Creates a child directory within the current directory + /// </summary> + /// <param name="directoryName">The name of the child directory</param> + /// <returns>A new <see cref="IsolatedStorageDirectory"/> for which <see cref="IsolatedStorageFileStream"/>s can be opened/created</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + public IsolatedStorageDirectory CreateChildDirectory(string directoryName) + { + return new IsolatedStorageDirectory(this, directoryName); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/SlidingWindowBufferExtensions.cs b/Utils/src/IO/SlidingWindowBufferExtensions.cs new file mode 100644 index 0000000..0509061 --- /dev/null +++ b/Utils/src/IO/SlidingWindowBufferExtensions.cs @@ -0,0 +1,213 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SlidingWindowBufferExtensions.cs +* +* SlidingWindowBufferExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; +using System.Runtime.CompilerServices; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Extention methods for <see cref="ISlindingWindowBuffer{T}"/> + /// </summary> + public static class SlidingWindowBufferExtensions + { + /// <summary> + /// Shifts/resets the current buffered data window down to the + /// begining of the buffer if the buffer window is shifted away + /// from the begining. + /// </summary> + /// <returns>The number of bytes of available space in the buffer</returns> + public static ERRNO CompactBufferWindow<T>(this ISlindingWindowBuffer<T> sBuf) + { + //Nothing to compact if the starting data pointer is at the beining of the window + if (sBuf.WindowStartPos > 0) + { + //Get span over engire buffer + Span<T> buffer = sBuf.Buffer.Span; + //Get data within window + Span<T> usedData = sBuf.Accumulated; + //Copy remaining to the begining of the buffer + usedData.CopyTo(buffer); + + //Reset positions, then advance to the specified size + sBuf.Reset(); + sBuf.Advance(usedData.Length); + } + //Return the number of bytes of available space + return sBuf.RemainingSize; + } + + /// <summary> + /// Appends the specified data to the end of the buffer + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="sBuf"></param> + /// <param name="val">The value to append to the end of the buffer</param> + /// <exception cref="IndexOutOfRangeException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Append<T>(this IDataAccumulator<T> sBuf, T val) + { + //Set the value at first position + sBuf.Remaining[0] = val; + //Advance by 1 + sBuf.Advance(1); + } + /// <summary> + /// Appends the specified data to the end of the buffer + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="sBuf"></param> + /// <param name="val">The value to append to the end of the buffer</param> + /// <exception cref="ArgumentException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Append<T>(this IDataAccumulator<T> sBuf, ReadOnlySpan<T> val) + { + val.CopyTo(sBuf.Remaining); + sBuf.Advance(val.Length); + } + /// <summary> + /// Formats and appends a value type to the accumulator with proper endianess + /// </summary> + /// <typeparam name="T">The value type to appent</typeparam> + /// <param name="accumulator">The binary accumulator to append</param> + /// <param name="value">The value type to append</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Append<T>(this IDataAccumulator<byte> accumulator, T value) where T: unmanaged + { + //Use forward reader for the memory extension to append a value type to a binary accumulator + ForwardOnlyWriter<byte> w = new(accumulator.Remaining); + w.Append(value); + accumulator.Advance(w.Written); + } + + /// <summary> + /// Attempts to write as much data as possible to the remaining space + /// in the buffer and returns the number of bytes accumulated. + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="accumulator"></param> + /// <param name="value">The value to accumulate</param> + /// <returns>The number of bytes accumulated</returns> + public static ERRNO TryAccumulate<T>(this IDataAccumulator<T> accumulator, ReadOnlySpan<T> value) + { + //Calc data size and reserve space for final crlf + int dataToCopy = Math.Min(value.Length, accumulator.RemainingSize); + + //Write as much data as possible + accumulator.Append(value[..dataToCopy]); + + //Return number of bytes not written + return dataToCopy; + } + + /// <summary> + /// Appends a <see cref="ISpanFormattable"/> instance to the end of the accumulator + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="accumulator"></param> + /// <param name="formattable">The formattable instance to write to the accumulator</param> + /// <param name="format">The format arguments</param> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Append<T>(this IDataAccumulator<char> accumulator, in T formattable, ReadOnlySpan<char> format = default) where T : struct, ISpanFormattable + { + ForwardOnlyWriter<char> writer = new(accumulator.Remaining); + writer.Append(formattable, format); + accumulator.Advance(writer.Written); + } + + /// <summary> + /// Uses the remaining data buffer to compile a <see cref="IStringSerializeable"/> + /// instance, then advances the accumulator by the number of characters used. + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="accumulator"></param> + /// <param name="compileable">The <see cref="IStringSerializeable"/> instance to compile</param> + public static void Append<T>(this IDataAccumulator<char> accumulator, in T compileable) where T : IStringSerializeable + { + //Write directly to the remaining space + int written = compileable.Compile(accumulator.Remaining); + //Advance the writer + accumulator.Advance(written); + } + + /// <summary> + /// Reads available data from the current window and writes as much as possible it to the supplied buffer + /// and advances the buffer window + /// </summary> + /// <typeparam name="T">Element type</typeparam> + /// <param name="sBuf"></param> + /// <param name="buffer">The output buffer to write data to</param> + /// <returns>The number of elements written to the buffer</returns> + public static ERRNO Read<T>(this ISlindingWindowBuffer<T> sBuf, in Span<T> buffer) + { + //Calculate the amount of data to copy + int dataToCopy = Math.Min(buffer.Length, sBuf.AccumulatedSize); + //Copy the data to the buffer + sBuf.Accumulated[..dataToCopy].CopyTo(buffer); + //Advance the window + sBuf.AdvanceStart(dataToCopy); + //Return the number of bytes copied + return dataToCopy; + } + + /// <summary> + /// Fills the remaining window space of the current accumulator with + /// data from the specified stream asynchronously. + /// </summary> + /// <param name="accumulator"></param> + /// <param name="input">The stream to read data from</param> + /// <param name="cancellationToken">A token to cancel the operation</param> + /// <returns>A value task representing the operation</returns> + public static async ValueTask AccumulateDataAsync(this ISlindingWindowBuffer<byte> accumulator, Stream input, CancellationToken cancellationToken) + { + //Get a buffer from the end of the current window to the end of the buffer + Memory<byte> bufWindow = accumulator.RemainingBuffer; + //Read from stream async + int read = await input.ReadAsync(bufWindow, cancellationToken); + //Update the end of the buffer window to the end of the read data + accumulator.Advance(read); + } + /// <summary> + /// Fills the remaining window space of the current accumulator with + /// data from the specified stream. + /// </summary> + /// <param name="accumulator"></param> + /// <param name="input">The stream to read data from</param> + public static void AccumulateData(this IDataAccumulator<byte> accumulator, Stream input) + { + //Get a buffer from the end of the current window to the end of the buffer + Span<byte> bufWindow = accumulator.Remaining; + //Read from stream async + int read = input.Read(bufWindow); + //Update the end of the buffer window to the end of the read data + accumulator.Advance(read); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/TemporayIsolatedFile.cs b/Utils/src/IO/TemporayIsolatedFile.cs new file mode 100644 index 0000000..3bee92b --- /dev/null +++ b/Utils/src/IO/TemporayIsolatedFile.cs @@ -0,0 +1,57 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: TemporayIsolatedFile.cs +* +* TemporayIsolatedFile.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.IO.IsolatedStorage; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Allows for temporary files to be generated, used, then removed from an <see cref="IsolatedStorageFile"/> + /// </summary> + public sealed class TemporayIsolatedFile : BackingStream<IsolatedStorageFileStream> + { + private readonly IsolatedStorageDirectory Storage; + private readonly string Filename; + /// <summary> + /// Creates a new temporary filestream within the specified <see cref="IsolatedStorageFile"/> + /// </summary> + /// <param name="storage">The file store to genreate temporary files within</param> + public TemporayIsolatedFile(IsolatedStorageDirectory storage) + { + //Store ref + this.Storage = storage; + //Creaet a new random filename + this.Filename = Path.GetRandomFileName(); + //try to created a new file within the isolaged storage + this.BaseStream = storage.CreateFile(this.Filename); + } + protected override void OnClose() + { + //Remove the file from the storage + Storage.DeleteFile(this.Filename); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/VnMemoryStream.cs b/Utils/src/IO/VnMemoryStream.cs new file mode 100644 index 0000000..389a7da --- /dev/null +++ b/Utils/src/IO/VnMemoryStream.cs @@ -0,0 +1,469 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnMemoryStream.cs +* +* VnMemoryStream.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using System.Runtime.InteropServices; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.IO +{ + + using Utils.Memory; + + /// <summary> + /// Provides an unmanaged memory stream. Desigend to help reduce garbage collector load for + /// high frequency memory operations. Similar to <see cref="UnmanagedMemoryStream"/> + /// </summary> + public sealed class VnMemoryStream : Stream, ICloneable + { + private long _position; + private long _length; + //Memory + private readonly MemoryHandle<byte> _buffer; + private bool IsReadonly; + //Default owns handle + private readonly bool OwnsHandle = true; + + /// <summary> + /// Creates a new <see cref="VnMemoryStream"/> pointing to the begining of memory, and consumes the handle. + /// </summary> + /// <param name="handle"><see cref="MemoryHandle{T}"/> to consume</param> + /// <param name="length">Length of the stream</param> + /// <param name="readOnly">Should the stream be readonly?</param> + /// <exception cref="ArgumentException"></exception> + /// <returns>A <see cref="VnMemoryStream"/> wrapper to access the handle data</returns> + public static VnMemoryStream ConsumeHandle(MemoryHandle<byte> handle, Int64 length, bool readOnly) + { + handle.ThrowIfClosed(); + return new VnMemoryStream(handle, length, readOnly, true); + } + + /// <summary> + /// Converts a writable <see cref="VnMemoryStream"/> to readonly to allow shallow copies + /// </summary> + /// <param name="stream">The stream to make readonly</param> + /// <returns>The readonly stream</returns> + public static VnMemoryStream CreateReadonly(VnMemoryStream stream) + { + //Set the readonly flag + stream.IsReadonly = true; + //Return the stream + return stream; + } + + /// <summary> + /// Creates a new memory stream + /// </summary> + public VnMemoryStream() : this(Memory.Shared) { } + /// <summary> + /// Create a new memory stream where buffers will be allocated from the specified heap + /// </summary> + /// <param name="heap"><see cref="PrivateHeap"/> to allocate memory from</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + public VnMemoryStream(IUnmangedHeap heap) : this(heap, 0, false) { } + + /// <summary> + /// Creates a new memory stream and pre-allocates the internal + /// buffer of the specified size on the specified heap to avoid resizing. + /// </summary> + /// <param name="heap"><see cref="PrivateHeap"/> to allocate memory from</param> + /// <param name="bufferSize">Number of bytes (length) of the stream if known</param> + /// <param name="zero">Zero memory allocations during buffer expansions</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public VnMemoryStream(IUnmangedHeap heap, long bufferSize, bool zero) + { + _ = heap ?? throw new ArgumentNullException(nameof(heap)); + _buffer = heap.Alloc<byte>(bufferSize, zero); + } + + /// <summary> + /// Creates a new memory stream from the data provided + /// </summary> + /// <param name="heap"><see cref="PrivateHeap"/> to allocate memory from</param> + /// <param name="data">Initial data</param> + public VnMemoryStream(IUnmangedHeap heap, ReadOnlySpan<byte> data) + { + _ = heap ?? throw new ArgumentNullException(nameof(heap)); + //Alloc the internal buffer to match the data stream + _buffer = heap.AllocAndCopy(data); + //Set length + _length = data.Length; + //Position will default to 0 cuz its dotnet :P + return; + } + + /// <summary> + /// WARNING: Dangerous constructor, make sure read-only and owns hanlde are set accordingly + /// </summary> + /// <param name="buffer">The buffer to referrence directly</param> + /// <param name="length">The length property of the stream</param> + /// <param name="readOnly">Is the stream readonly (should mostly be true!)</param> + /// <param name="ownsHandle">Does the new stream own the memory -> <paramref name="buffer"/></param> + private VnMemoryStream(MemoryHandle<byte> buffer, long length, bool readOnly, bool ownsHandle) + { + OwnsHandle = ownsHandle; + _buffer = buffer; //Consume the handle + _length = length; //Store length of the buffer + IsReadonly = readOnly; + } + + /// <summary> + /// UNSAFE Number of bytes between position and length. Never negative + /// </summary> + private long LenToPosDiff => Math.Max(_length - _position, 0); + + /// <summary> + /// If the current stream is a readonly stream, creates an unsafe shallow copy for reading only. + /// </summary> + /// <returns>New stream shallow copy of the internal stream</returns> + /// <exception cref="NotSupportedException"></exception> + public VnMemoryStream GetReadonlyShallowCopy() + { + //Create a new readonly copy (stream does not own the handle) + return !IsReadonly + ? throw new NotSupportedException("This stream is not readonly. Cannot create shallow copy on a mutable stream") + : new VnMemoryStream(_buffer, _length, true, false); + } + + /// <summary> + /// Writes data directly to the destination stream from the internal buffer + /// without allocating or copying any data. + /// </summary> + /// <param name="destination">The stream to write data to</param> + /// <param name="bufferSize">The size of the chunks to write to the destination stream</param> + /// <exception cref="IOException"></exception> + public override void CopyTo(Stream destination, int bufferSize) + { + _ = destination ?? throw new ArgumentNullException(nameof(destination)); + + if (!destination.CanWrite) + { + throw new IOException("The destinaion stream is not writeable"); + } + + do + { + //Calc the remaining bytes to read no larger than the buffer size + int bytesToRead = (int)Math.Min(LenToPosDiff, bufferSize); + + //Create a span wrapper by using the offet function to support memory handles larger than 2gb + ReadOnlySpan<byte> span = _buffer.GetOffsetSpan(_position, bytesToRead); + + destination.Write(span); + + //Update position + _position += bytesToRead; + + } while (LenToPosDiff > 0); + } + + /// <summary> + /// Allocates a temporary buffer of the desired size, copies data from the internal + /// buffer and writes it to the destination buffer asynchronously. + /// </summary> + /// <param name="destination">The stream to write output data to</param> + /// <param name="bufferSize">The size of the buffer to use when copying data</param> + /// <param name="cancellationToken">A token to cancel the opreation</param> + /// <returns>A task that resolves when the remaining data in the stream has been written to the destination</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public override async Task CopyToAsync(Stream destination, int bufferSize, CancellationToken cancellationToken) + { + _ = destination ?? throw new ArgumentNullException(nameof(destination)); + + if (!destination.CanWrite) + { + throw new IOException("The destinaion stream is not writeable"); + } + + cancellationToken.ThrowIfCancellationRequested(); + + /* + * Alloc temp copy buffer. This is a requirement because + * the stream may be larger than an int32 so it must be + * copied by segment + */ + + using VnTempBuffer<byte> copyBuffer = new(bufferSize); + + do + { + //read from internal stream + int read = Read(copyBuffer); + + if(read <= 0) + { + break; + } + + //write async + await destination.WriteAsync(copyBuffer.AsMemory(0, read), cancellationToken); + + } while (true); + + } + + /// <summary> + /// <inheritdoc/> + /// <para> + /// This property is always true + /// </para> + /// </summary> + public override bool CanRead => true; + /// <summary> + /// <inheritdoc/> + /// <para> + /// This propery is always true + /// </para> + /// </summary> + public override bool CanSeek => true; + /// <summary> + /// True unless the stream is (or has been converted to) a readonly + /// stream. + /// </summary> + public override bool CanWrite => !IsReadonly; + ///<inheritdoc/> + public override long Length => _length; + ///<inheritdoc/> + public override bool CanTimeout => false; + + ///<inheritdoc/> + public override long Position + { + get => _position; + set => Seek(value, SeekOrigin.Begin); + } + /// <summary> + /// Closes the stream and frees the internal allocated memory blocks + /// </summary> + public override void Close() + { + //Only dispose buffer if we own it + if (OwnsHandle) + { + _buffer.Dispose(); + } + } + ///<inheritdoc/> + public override void Flush() { } + // Override to reduce base class overhead + ///<inheritdoc/> + public override Task FlushAsync(CancellationToken cancellationToken) => Task.CompletedTask; + ///<inheritdoc/> + public override int Read(byte[] buffer, int offset, int count) => Read(new Span<byte>(buffer, offset, count)); + ///<inheritdoc/> + public override int Read(Span<byte> destination) + { + if (destination.Length == 0) + { + return 0; + } + //Number of bytes to read from memory buffer + int bytesToRead = checked((int)Math.Min(LenToPosDiff, destination.Length)); + //Copy bytes to buffer + Memory.Copy(_buffer, _position, destination, 0, bytesToRead); + //Increment buffer position + _position += bytesToRead; + //Bytestoread should never be larger than int.max because span length is an integer + return bytesToRead; + } + + /* + * Async reading will always run synchronously in a memory stream, + * so overrides are just so avoid base class overhead + */ + ///<inheritdoc/> + public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default) + { + //Read synchronously and return a completed task + int read = Read(buffer.Span); + return ValueTask.FromResult(read); + } + ///<inheritdoc/> + public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + //Read synchronously and return a completed task + int read = Read(buffer.AsSpan(offset, count)); + return Task.FromResult(read); + } + ///<inheritdoc/> + public override long Seek(long offset, SeekOrigin origin) + { + if (offset < 0) + { + throw new ArgumentOutOfRangeException(nameof(offset), "Offset cannot be less than 0"); + } + switch (origin) + { + case SeekOrigin.Begin: + //Length will never be greater than int.Max so output will never exceed int.max + _position = Math.Min(_length, offset); + return _position; + case SeekOrigin.Current: + long newPos = _position + offset; + //Length will never be greater than int.Max so output will never exceed length + _position = Math.Min(_length, newPos); + return newPos; + case SeekOrigin.End: + long real_index = _length - offset; + //If offset moves the position negative, just set the position to 0 and continue + _position = Math.Min(real_index, 0); + return real_index; + default: + throw new ArgumentException("Stream operation is not supported on current stream"); + } + } + + + /// <summary> + /// Resizes the internal buffer to the exact size (in bytes) of the + /// value argument. A value of 0 will free the entire buffer. A value + /// greater than zero will resize the buffer (and/or alloc) + /// </summary> + /// <param name="value">The size of the stream (and internal buffer)</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="NotSupportedException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public override void SetLength(long value) + { + if (IsReadonly) + { + throw new NotSupportedException("This stream is readonly"); + } + if (value < 0) + { + throw new ArgumentOutOfRangeException(nameof(value), "Value cannot be less than 0"); + } + //Resize the buffer to the specified length + _buffer.Resize(value); + //Set length + _length = value; + //Make sure the position is not pointing outside of the buffer + _position = Math.Min(_position, _length); + return; + } + ///<inheritdoc/> + public override void Write(byte[] buffer, int offset, int count) => Write(new ReadOnlySpan<byte>(buffer, offset, count)); + ///<inheritdoc/> + public override void Write(ReadOnlySpan<byte> buffer) + { + if (IsReadonly) + { + throw new NotSupportedException("Write operation is not allowed on readonly stream!"); + } + //Calculate the new final position + long newPos = (_position + buffer.Length); + //Determine if the buffer needs to be expanded + if (buffer.Length > LenToPosDiff) + { + //Expand buffer if required + _buffer.ResizeIfSmaller(newPos); + //Update length + _length = newPos; + } + //Copy the input buffer to the internal buffer + Memory.Copy(buffer, _buffer, _position); + //Update the position + _position = newPos; + return; + } + ///<inheritdoc/> + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + //Write synchronously and return a completed task + Write(buffer, offset, count); + return Task.CompletedTask; + } + ///<inheritdoc/> + public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default) + { + //Write synchronously and return a completed task + Write(buffer.Span); + return ValueTask.CompletedTask; + } + ///<inheritdoc/> + public override void WriteByte(byte value) + { + Span<byte> buf = MemoryMarshal.CreateSpan(ref value, 1); + Write(buf); + } + + /// <summary> + /// Allocates and copies internal buffer to new managed byte[] + /// </summary> + /// <returns>Copy of internal buffer</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public byte[] ToArray() + { + //Alloc a new array of the size of the internal buffer + byte[] data = new byte[_length]; + //Copy data from the internal buffer to the output buffer + _buffer.Span.CopyTo(data); + return data; + + } + /// <summary> + /// Returns a <see cref="ReadOnlySpan{T}"/> window over the data within the entire stream + /// </summary> + /// <returns>A <see cref="ReadOnlySpan{T}"/> of the data within the entire stream</returns> + /// <exception cref="OverflowException"></exception> + public ReadOnlySpan<byte> AsSpan() + { + ReadOnlySpan<byte> output = _buffer.Span; + return output[..(int)_length]; + } + + /// <summary> + /// If the current stream is a readonly stream, creates a shallow copy for reading only. + /// </summary> + /// <returns>New stream shallow copy of the internal stream</returns> + /// <exception cref="InvalidOperationException"></exception> + public object Clone() => GetReadonlyShallowCopy(); + + /* + * Override the Dispose async method to avoid the base class overhead + * and task allocation since this will always be a syncrhonous + * operation (freeing memory) + */ + + ///<inheritdoc/> + public override ValueTask DisposeAsync() + { + //Dispose and return completed task + base.Dispose(true); + GC.SuppressFinalize(this); + return ValueTask.CompletedTask; + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/VnStreamReader.cs b/Utils/src/IO/VnStreamReader.cs new file mode 100644 index 0000000..70b9734 --- /dev/null +++ b/Utils/src/IO/VnStreamReader.cs @@ -0,0 +1,180 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnStreamReader.cs +* +* VnStreamReader.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Binary based buffered text reader, optimized for reading network streams + /// </summary> + public class VnStreamReader : TextReader, IVnTextReader + { + private bool disposedValue; + + private readonly ISlindingWindowBuffer<byte> _buffer; + ///<inheritdoc/> + public virtual Stream BaseStream { get; } + ///<inheritdoc/> + public Encoding Encoding { get; } + + /// <summary> + /// Number of available bytes of buffered data within the current buffer window + /// </summary> + public int Available => _buffer.AccumulatedSize; + /// <summary> + /// Gets or sets the line termination used to deliminate a line of data + /// </summary> + public ReadOnlyMemory<byte> LineTermination { get; set; } + Span<byte> IVnTextReader.BufferedDataWindow => _buffer.Accumulated; + + /// <summary> + /// Creates a new <see cref="TextReader"/> that reads encoded data from the base. + /// Internal buffers will be alloced from <see cref="ArrayPool{T}.Shared"/> + /// </summary> + /// <param name="baseStream">The underlying stream to read data from</param> + /// <param name="enc">The <see cref="Encoding"/> to use when reading from the stream</param> + /// <param name="bufferSize">The size of the internal binary buffer</param> + public VnStreamReader(Stream baseStream, Encoding enc, int bufferSize) + { + BaseStream = baseStream; + Encoding = enc; + //Init a new buffer + _buffer = InitializeBuffer(bufferSize); + } + + /// <summary> + /// Invoked by the constuctor method to allocte the internal buffer with the specified buffer size. + /// </summary> + /// <param name="bufferSize">The requested size of the buffer to alloc</param> + /// <remarks>By default requests the buffer from the <see cref="ArrayPool{T}.Shared"/> instance</remarks> + protected virtual ISlindingWindowBuffer<byte> InitializeBuffer(int bufferSize) => new ArrayPoolStreamBuffer<byte>(ArrayPool<byte>.Shared, bufferSize); + + ///<inheritdoc/> + public override async Task<string?> ReadLineAsync() + { + //If buffered data is available, check for line termination + if (Available > 0) + { + //Get current buffer window + Memory<byte> buffered = _buffer.AccumulatedBuffer; + //search for line termination in current buffer + int term = buffered.IndexOf(LineTermination); + //Termination found in buffer window + if (term > -1) + { + //Capture the line from the begining of the window to the termination + Memory<byte> line = buffered[..term]; + //Shift the window to the end of the line (excluding the termination) + _buffer.AdvanceStart(term + LineTermination.Length); + //Decode the line to a string + return Encoding.GetString(line.Span); + } + //Termination not found + } + //Compact the buffer window and see if space is avialble to buffer more data + if (_buffer.CompactBufferWindow()) + { + //There is room, so buffer more data + await _buffer.AccumulateDataAsync(BaseStream, CancellationToken.None); + //Check again to see if more data is buffered + if (Available <= 0) + { + //No string found + return null; + } + //Get current buffer window + Memory<byte> buffered = _buffer.AccumulatedBuffer; + //search for line termination in current buffer + int term = buffered.IndexOf(LineTermination); + //Termination found in buffer window + if (term > -1) + { + //Capture the line from the begining of the window to the termination + Memory<byte> line = buffered[..term]; + //Shift the window to the end of the line (excluding the termination) + _buffer.AdvanceStart(term + LineTermination.Length); + //Decode the line to a string + return Encoding.GetString(line.Span); + } + } + //Termination not found within the entire buffer, so buffer space has been exhausted + + //OOM is raised in the TextReader base class, the standard is preserved +#pragma warning disable CA2201 // Do not raise reserved exception types + throw new OutOfMemoryException("A line termination was not found within the buffer"); +#pragma warning restore CA2201 // Do not raise reserved exception types + } + + ///<inheritdoc/> + public override int Read(char[] buffer, int index, int count) => Read(buffer.AsSpan(index, count)); + ///<inheritdoc/> + public override int Read(Span<char> buffer) + { + if (Available <= 0) + { + return 0; + } + //Get current buffer window + Span<byte> buffered = _buffer.Accumulated; + //Convert all avialable data + int encoded = Encoding.GetChars(buffered, buffer); + //Shift buffer window to the end of the converted data + _buffer.AdvanceStart(encoded); + //return the number of chars written + return Encoding.GetCharCount(buffered); + } + ///<inheritdoc/> + public override void Close() => _buffer.Close(); + ///<inheritdoc/> + protected override void Dispose(bool disposing) + { + if (!disposedValue) + { + Close(); + disposedValue = true; + } + base.Dispose(disposing); + } + + /// <summary> + /// Resets the internal buffer window + /// </summary> + protected void ClearBuffer() + { + _buffer.Reset(); + } + + void IVnTextReader.Advance(int count) => _buffer.AdvanceStart(count); + void IVnTextReader.FillBuffer() => _buffer.AccumulateData(BaseStream); + ERRNO IVnTextReader.CompactBufferWindow() => _buffer.CompactBufferWindow(); + } +}
\ No newline at end of file diff --git a/Utils/src/IO/VnStreamWriter.cs b/Utils/src/IO/VnStreamWriter.cs new file mode 100644 index 0000000..37d700c --- /dev/null +++ b/Utils/src/IO/VnStreamWriter.cs @@ -0,0 +1,292 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnStreamWriter.cs +* +* VnStreamWriter.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Memory; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Provides a memory optimized <see cref="TextWriter"/> implementation. Optimized for writing + /// to network streams + /// </summary> + public class VnStreamWriter : TextWriter + { + private readonly Encoder Enc; + + private readonly ISlindingWindowBuffer<byte> _buffer; + + private bool closed; + + /// <summary> + /// Gets the underlying stream that interfaces with the backing store + /// </summary> + public virtual Stream BaseStream { get; } + ///<inheritdoc/> + public override Encoding Encoding { get; } + + /// <summary> + /// Line termination to use when writing lines to the output + /// </summary> + public ReadOnlyMemory<byte> LineTermination { get; set; } + ///<inheritdoc/> + public override string NewLine + { + get => Encoding.GetString(LineTermination.Span); + set => LineTermination = Encoding.GetBytes(value); + } + + /// <summary> + /// Creates a new <see cref="VnStreamWriter"/> that writes formatted data + /// to the specified base stream + /// </summary> + /// <param name="baseStream">The stream to write data to</param> + /// <param name="encoding">The <see cref="Encoding"/> to use when writing data</param> + /// <param name="bufferSize">The size of the internal buffer used to buffer binary data before writing to the base stream</param> + public VnStreamWriter(Stream baseStream, Encoding encoding, int bufferSize = 1024) + { + //Store base stream + BaseStream = baseStream; + Encoding = encoding; + //Get an encoder + Enc = encoding.GetEncoder(); + _buffer = InitializeBuffer(bufferSize); + } + + /// <summary> + /// Invoked by the constuctor method to allocte the internal buffer with the specified buffer size. + /// </summary> + /// <param name="bufferSize">The requested size of the buffer to alloc</param> + /// <remarks>By default requests the buffer from the <see cref="MemoryPool{T}.Shared"/> instance</remarks> + protected virtual ISlindingWindowBuffer<byte> InitializeBuffer(int bufferSize) => new ArrayPoolStreamBuffer<byte>(ArrayPool<byte>.Shared, bufferSize); + ///<inheritdoc/> + public void Write(byte value) + { + //See if there is room in the binary buffer + if (_buffer.AccumulatedSize == 0) + { + //There is not enough room to store the single byte + Flush(); + } + //Store at the end of the window + _buffer.Append(value); + } + ///<inheritdoc/> + public override void Write(char value) + { + ReadOnlySpan<char> tbuf = MemoryMarshal.CreateSpan(ref value, 0x01); + Write(tbuf); + } + ///<inheritdoc/> + public override void Write(object value) => Write(value.ToString()); + ///<inheritdoc/> + public override void Write(string value) => Write(value.AsSpan()); + ///<inheritdoc/> + public override void Write(ReadOnlySpan<char> buffer) + { + Check(); + + ForwardOnlyReader<char> reader = new(buffer); + + //Create a variable for a character buffer window + bool completed; + do + { + //Get an available buffer window to store characters in and convert the characters to binary + Enc.Convert(reader.Window, _buffer.Remaining, true, out int charsUsed, out int bytesUsed, out completed); + //Update byte position + _buffer.Advance(bytesUsed); + //Update char position + reader.Advance(charsUsed); + + //Converting did not complete because the buffer was too small + if (!completed || reader.WindowSize == 0) + { + //Flush the buffer and continue + Flush(); + } + + } while (!completed); + //Reset the encoder + Enc.Reset(); + } + ///<inheritdoc/> + public override async Task WriteAsync(ReadOnlyMemory<char> buffer, CancellationToken cancellationToken = default) + { + Check(); + //Create a variable for a character buffer window + bool completed; + ForwardOnlyMemoryReader<char> reader = new(buffer); + do + { + //Get an available buffer window to store characters in and convert the characters to binary + Enc.Convert(reader.Window.Span, _buffer.Remaining, true, out int charsUsed, out int bytesUsed, out completed); + //Update byte position + _buffer.Advance(bytesUsed); + //Update char position + reader.Advance(charsUsed); + //Converting did not complete because the buffer was too small + if (!completed || reader.WindowSize == 0) + { + //Flush the buffer and continue + await FlushWriterAsync(cancellationToken); + } + } while (!completed); + //Reset the encoder + Enc.Reset(); + } + + ///<inheritdoc/> + public override void WriteLine() + { + Check(); + //See if there is room in the binary buffer + if (_buffer.RemainingSize < LineTermination.Length) + { + //There is not enough room to store the termination, so we need to flush the buffer + Flush(); + } + _buffer.Append(LineTermination.Span); + } + ///<inheritdoc/> + public override void WriteLine(object value) => WriteLine(value.ToString()); + ///<inheritdoc/> + public override void WriteLine(string value) => WriteLine(value.AsSpan()); + ///<inheritdoc/> + public override void WriteLine(ReadOnlySpan<char> buffer) + { + //Write the value itself + Write(buffer); + //Write the line termination + WriteLine(); + } + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public override void Flush() + { + Check(); + //If data is available to be written, write it to the base stream + if (_buffer.AccumulatedSize > 0) + { + //Write all buffered data to stream + BaseStream.Write(_buffer.Accumulated); + //Reset the buffer + _buffer.Reset(); + } + } + /// <summary> + /// Asynchronously flushes the internal buffers to the <see cref="BaseStream"/>, and resets the internal buffer state + /// </summary> + /// <returns>A <see cref="ValueTask"/> that represents the asynchronous flush operation</returns> + /// <exception cref="ObjectDisposedException"></exception> + public async ValueTask FlushWriterAsync(CancellationToken cancellationToken = default) + { + Check(); + if (_buffer.AccumulatedSize > 0) + { + //Flush current window to the stream + await BaseStream.WriteAsync(_buffer.AccumulatedBuffer, cancellationToken); + //Reset the buffer + _buffer.Reset(); + } + } + + ///<inheritdoc/> + public override Task FlushAsync() => FlushWriterAsync().AsTask(); + + /// <summary> + /// Resets internal properies for resuse + /// </summary> + protected void Reset() + { + _buffer.Reset(); + Enc.Reset(); + } + ///<inheritdoc/> + public override void Close() + { + //Only invoke close once + if (closed) + { + return; + } + try + { + Flush(); + } + finally + { + //Release the memory handle if its set + _buffer.Close(); + //Set closed flag + closed = true; + } + } + ///<inheritdoc/> + protected override void Dispose(bool disposing) + { + Close(); + base.Dispose(disposing); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private void Check() + { + if (closed) + { + throw new ObjectDisposedException("The stream is closed"); + } + } + ///<inheritdoc/> + public override async ValueTask DisposeAsync() + { + //Only invoke close once + if (closed) + { + return; + } + try + { + await FlushWriterAsync(); + } + finally + { + //Set closed flag + closed = true; + //Release the memory handle if its set + _buffer.Close(); + } + GC.SuppressFinalize(this); + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/VnTextReaderExtensions.cs b/Utils/src/IO/VnTextReaderExtensions.cs new file mode 100644 index 0000000..119461b --- /dev/null +++ b/Utils/src/IO/VnTextReaderExtensions.cs @@ -0,0 +1,223 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnTextReaderExtensions.cs +* +* VnTextReaderExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// Extension methods to help reuse code for used TextReader implementations + /// </summary> + public static class VnTextReaderExtensions + { + public const int E_BUFFER_TOO_SMALL = -1; + + + /* + * Generic extensions provide constained compiler method invocation + * for structs the implement the IVNtextReader + */ + + /// <summary> + /// Attempts to read a line from the stream and store it in the specified buffer + /// </summary> + /// <param name="reader"></param> + /// <param name="charBuffer">The character buffer to write data to</param> + /// <returns>Returns the number of bytes read, <see cref="E_BUFFER_TOO_SMALL"/> + /// if the buffer was not large enough, 0 if no data was available</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <remarks>Allows reading lines of data from the stream without allocations</remarks> + public static ERRNO ReadLine<T>(this ref T reader, Span<char> charBuffer) where T:struct, IVnTextReader + { + return readLine(ref reader, charBuffer); + } + /// <summary> + /// Attempts to read a line from the stream and store it in the specified buffer + /// </summary> + /// <param name="reader"></param> + /// <param name="charBuffer">The character buffer to write data to</param> + /// <returns>Returns the number of bytes read, <see cref="E_BUFFER_TOO_SMALL"/> + /// if the buffer was not large enough, 0 if no data was available</returns> + /// <exception cref="OutOfMemoryException"></exception> + /// <remarks>Allows reading lines of data from the stream without allocations</remarks> + public static ERRNO ReadLine<T>(this T reader, Span<char> charBuffer) where T : class, IVnTextReader + { + return readLine(ref reader, charBuffer); + } + + /// <summary> + /// Fill a buffer with reamining buffered data + /// </summary> + /// <param name="reader"></param> + /// <param name="buffer">Buffer to copy data to</param> + /// <param name="offset">Offset in buffer to begin writing</param> + /// <param name="count">Number of bytes to read</param> + /// <returns>The number of bytes copied to the input buffer</returns> + public static int ReadRemaining<T>(this ref T reader, byte[] buffer, int offset, int count) where T : struct, IVnTextReader + { + return reader.ReadRemaining(buffer.AsSpan(offset, count)); + } + /// <summary> + /// Fill a buffer with reamining buffered data + /// </summary> + /// <param name="reader"></param> + /// <param name="buffer">Buffer to copy data to</param> + /// <param name="offset">Offset in buffer to begin writing</param> + /// <param name="count">Number of bytes to read</param> + /// <returns>The number of bytes copied to the input buffer</returns> + public static int ReadRemaining<T>(this T reader, byte[] buffer, int offset, int count) where T : class, IVnTextReader + { + return reader.ReadRemaining(buffer.AsSpan(offset, count)); + } + + /// <summary> + /// Fill a buffer with reamining buffered data, up to + /// the size of the supplied buffer + /// </summary> + /// <param name="reader"></param> + /// <param name="buffer">Buffer to copy data to</param> + /// <returns>The number of bytes copied to the input buffer</returns> + /// <remarks>You should use the <see cref="IVnTextReader.Available"/> property to know how much remaining data is buffered</remarks> + public static int ReadRemaining<T>(this ref T reader, Span<byte> buffer) where T : struct, IVnTextReader + { + return readRemaining(ref reader, buffer); + } + /// <summary> + /// Fill a buffer with reamining buffered data, up to + /// the size of the supplied buffer + /// </summary> + /// <param name="reader"></param> + /// <param name="buffer">Buffer to copy data to</param> + /// <returns>The number of bytes copied to the input buffer</returns> + /// <remarks>You should use the <see cref="IVnTextReader.Available"/> property to know how much remaining data is buffered</remarks> + public static int ReadRemaining<T>(this T reader, Span<byte> buffer) where T : class, IVnTextReader + { + return readRemaining(ref reader, buffer); + } + + private static ERRNO readLine<T>(ref T reader, Span<char> chars) where T: IVnTextReader + { + /* + * I am aware of a potential bug, the line decoding process + * shifts the interal buffer by the exact number of bytes to + * the end of the line, without considering if the decoder failed + * to properly decode the entire line. + * + * I dont expect this to be an issue unless there is a bug within the specified + * encoder implementation + */ + ReadOnlySpan<byte> LineTermination = reader.LineTermination.Span; + //If buffered data is available, check for line termination + if (reader.Available > 0) + { + //Get current buffer window + ReadOnlySpan<byte> bytes = reader.BufferedDataWindow; + //search for line termination in current buffer + int term = bytes.IndexOf(LineTermination); + //Termination found in buffer window + if (term > -1) + { + //Capture the line from the begining of the window to the termination + ReadOnlySpan<byte> line = bytes[..term]; + //Get the number ot chars + int charCount = reader.Encoding.GetCharCount(line); + //See if the buffer is large enough + if (bytes.Length < charCount) + { + return E_BUFFER_TOO_SMALL; + } + //Use the decoder to convert the data + _ = reader.Encoding.GetChars(line, chars); + //Shift the window to the end of the line (excluding the termination, regardless of the conversion result) + reader.Advance(term + LineTermination.Length); + //Return the number of characters + return charCount; + } + //Termination not found but there may be more data waiting + } + //Compact the buffer window and make sure it was compacted so there is room to fill the buffer + if (reader.CompactBufferWindow()) + { + //There is room, so buffer more data + reader.FillBuffer(); + //Check again to see if more data is buffered + if (reader.Available <= 0) + { + //No data avialable + return 0; + } + //Get current buffer window + ReadOnlySpan<byte> bytes = reader.BufferedDataWindow; + //search for line termination in current buffer + int term = bytes.IndexOf(LineTermination); + //Termination found in buffer window + if (term > -1) + { + //Capture the line from the begining of the window to the termination + ReadOnlySpan<byte> line = bytes[..term]; + //Get the number ot chars + int charCount = reader.Encoding.GetCharCount(line); + //See if the buffer is large enough + if (bytes.Length < charCount) + { + return E_BUFFER_TOO_SMALL; + } + //Use the decoder to convert the data + _ = reader.Encoding.GetChars(line, chars); + //Shift the window to the end of the line (excluding the termination, regardless of the conversion result) + reader.Advance(term + LineTermination.Length); + //Return the number of characters + return charCount; + } + } + + //Termination not found within the entire buffer, so buffer space has been exhausted + + //Supress as this response is expected when the buffer is exhausted, +#pragma warning disable CA2201 // Do not raise reserved exception types + throw new OutOfMemoryException("The line was not found within the current buffer, cannot continue"); +#pragma warning restore CA2201 // Do not raise reserved exception types + } + + private static int readRemaining<T>(ref T reader, Span<byte> buffer) where T: IVnTextReader + { + //guard for empty buffer + if (buffer.Length == 0 || reader.Available == 0) + { + return 0; + } + //get the remaining bytes in the reader + Span<byte> remaining = reader.BufferedDataWindow; + //Calculate the number of bytes to copy + int canCopy = Math.Min(remaining.Length, buffer.Length); + //Copy remaining bytes to buffer + remaining[..canCopy].CopyTo(buffer); + //Shift the window by the number of bytes copied + reader.Advance(canCopy); + return canCopy; + } + } +}
\ No newline at end of file diff --git a/Utils/src/IO/WriteOnlyBufferedStream.cs b/Utils/src/IO/WriteOnlyBufferedStream.cs new file mode 100644 index 0000000..5e7faa1 --- /dev/null +++ b/Utils/src/IO/WriteOnlyBufferedStream.cs @@ -0,0 +1,255 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: WriteOnlyBufferedStream.cs +* +* WriteOnlyBufferedStream.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Threading; +using System.Threading.Tasks; + +using VNLib.Utils.Memory; + +namespace VNLib.Utils.IO +{ + /// <summary> + /// A basic accumulator style write buffered stream + /// </summary> + public class WriteOnlyBufferedStream : Stream + { + private readonly ISlindingWindowBuffer<byte> _buffer; + private readonly bool LeaveOpen; + + /// <summary> + /// Gets the underlying stream that interfaces with the backing store + /// </summary> + public Stream BaseStream { get; init; } + + /// <summary> + /// Initalizes a new <see cref="WriteOnlyBufferedStream"/> using the + /// specified backing stream, using the specified buffer size, and + /// optionally leaves the stream open + /// </summary> + /// <param name="baseStream">The backing stream to write buffered data to</param> + /// <param name="bufferSize">The size of the internal buffer</param> + /// <param name="leaveOpen">A value indicating of the stream should be left open when the buffered stream is closed</param> + public WriteOnlyBufferedStream(Stream baseStream, int bufferSize, bool leaveOpen = false) + { + BaseStream = baseStream; + //Create buffer + _buffer = InitializeBuffer(bufferSize); + LeaveOpen = leaveOpen; + } + /// <summary> + /// Invoked by the constuctor method to allocte the internal buffer with the specified buffer size. + /// </summary> + /// <param name="bufferSize">The requested size of the buffer to alloc</param> + /// <remarks>By default requests the buffer from the <see cref="ArrayPool{T}.Shared"/> instance</remarks> + protected virtual ISlindingWindowBuffer<byte> InitializeBuffer(int bufferSize) + { + return new ArrayPoolStreamBuffer<byte>(ArrayPool<byte>.Shared, bufferSize); + } + + ///<inheritdoc/> + public override void Close() + { + try + { + //Make sure the buffer is empty + WriteBuffer(); + + if (!LeaveOpen) + { + //Dispose stream + BaseStream.Dispose(); + } + } + finally + { + _buffer.Close(); + } + } + ///<inheritdoc/> + public override async ValueTask DisposeAsync() + { + try + { + if (_buffer.AccumulatedSize > 0) + { + await WriteBufferAsync(CancellationToken.None); + } + + if (!LeaveOpen) + { + //Dispose stream + await BaseStream.DisposeAsync(); + } + + GC.SuppressFinalize(this); + } + finally + { + _buffer.Close(); + } + } + + ///<inheritdoc/> + public override void Flush() => WriteBuffer(); + ///<inheritdoc/> + public override Task FlushAsync(CancellationToken cancellationToken) => WriteBufferAsync(cancellationToken).AsTask(); + + private void WriteBuffer() + { + //Only if data is available to write + if (_buffer.AccumulatedSize > 0) + { + //Write data to stream + BaseStream.Write(_buffer.Accumulated); + //Reset position + _buffer.Reset(); + } + } + + private async ValueTask WriteBufferAsync(CancellationToken token = default) + { + if(_buffer.AccumulatedSize > 0) + { + await BaseStream.WriteAsync(_buffer.AccumulatedBuffer, token); + _buffer.Reset(); + } + } + ///<inheritdoc/> + public override void Write(byte[] buffer, int offset, int count) => Write(buffer.AsSpan(offset, count)); + + public override void Write(ReadOnlySpan<byte> buffer) + { + ForwardOnlyReader<byte> reader = new(buffer); + //Attempt to buffer/flush data until all data is sent + do + { + //Try to buffer as much as possible + ERRNO buffered = _buffer.TryAccumulate(reader.Window); + + if(buffered < reader.WindowSize) + { + //Buffer is full and needs to be flushed + WriteBuffer(); + //Advance reader and continue to buffer + reader.Advance(buffered); + continue; + } + + break; + } + while (true); + } + + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + return WriteAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + + public async override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken = default) + { + ForwardOnlyMemoryReader<byte> reader = new(buffer); + //Attempt to buffer/flush data until all data is sent + do + { + //Try to buffer as much as possible + ERRNO buffered = _buffer.TryAccumulate(reader.Window.Span); + + if (buffered < reader.WindowSize) + { + //Buffer is full and needs to be flushed + await WriteBufferAsync(cancellationToken); + //Advance reader and continue to buffer + reader.Advance(buffered); + continue; + } + + break; + } + while (true); + } + + + /// <summary> + /// Always false + /// </summary> + public override bool CanRead => false; + /// <summary> + /// Always returns false + /// </summary> + public override bool CanSeek => false; + /// <summary> + /// Always true + /// </summary> + public override bool CanWrite => true; + /// <summary> + /// Returns the size of the underlying buffer + /// </summary> + public override long Length => _buffer.AccumulatedSize; + /// <summary> + /// Always throws <see cref="NotSupportedException"/> + /// </summary> + /// <exception cref="NotSupportedException"></exception> + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + /// <summary> + /// Always throws <see cref="NotSupportedException"/> + /// </summary> + /// <exception cref="NotSupportedException"></exception> + /// <returns></returns> + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException("This stream is not readable"); + } + + /// <summary> + /// Always throws <see cref="NotSupportedException"/> + /// </summary> + /// <exception cref="NotSupportedException"></exception> + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + /// <summary> + /// Always throws <see cref="NotSupportedException"/> + /// </summary> + /// <exception cref="NotSupportedException"></exception> + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + throw new NotImplementedException(); + } + + public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default) + { + throw new NotImplementedException(); + } + } +} diff --git a/Utils/src/IObjectStorage.cs b/Utils/src/IObjectStorage.cs new file mode 100644 index 0000000..5c99cd8 --- /dev/null +++ b/Utils/src/IObjectStorage.cs @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IObjectStorage.cs +* +* IObjectStorage.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Utils +{ + /// <summary> + /// This object will provide methods for storing and retreiving objects by key-value pairing + /// </summary> + public interface IObjectStorage + { + /// <summary> + /// Attempts to retrieve the specified object from storage + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="key">Key for storage</param> + /// <returns>The object in storage, or T.default if object is not found</returns> + public T GetObject<T>(string key); + + /// <summary> + /// Stores the specified object with the specified key + /// </summary> + /// <typeparam name="T"></typeparam> + /// <param name="key">Key paired with object</param> + /// <param name="obj">Object to store</param> + public void SetObject<T>(string key, T obj); + } +}
\ No newline at end of file diff --git a/Utils/src/Logging/ILogProvider.cs b/Utils/src/Logging/ILogProvider.cs new file mode 100644 index 0000000..55dbd6f --- /dev/null +++ b/Utils/src/Logging/ILogProvider.cs @@ -0,0 +1,79 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ILogProvider.cs +* +* ILogProvider.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Logging +{ + /// <summary> + /// Self-contained logging interface that allows for applications events to be written to an + /// output source + /// </summary> + public interface ILogProvider + { + /// <summary> + /// Flushes any buffers to the output source + /// </summary> + abstract void Flush(); + + /// <summary> + /// Writes the string to the log with the specified priority log level + /// </summary> + /// <param name="level">The log priority level</param> + /// <param name="value">The message to print</param> + void Write(LogLevel level, string value); + /// <summary> + /// Writes the exception and optional string to the log with the specified priority log level + /// </summary> + /// <param name="level">The log priority level</param> + /// <param name="exception">An exception object to write</param> + /// <param name="value">The message to print</param> + void Write(LogLevel level, Exception exception, string value = ""); + /// <summary> + /// Writes the template string and params arguments to the log with the specified priority log level + /// </summary> + /// <param name="level">The log priority level</param> + /// <param name="value">The log template string</param> + /// <param name="args">Variable length array of objects to log with the specified templatre</param> + void Write(LogLevel level, string value, params object?[] args); + /// <summary> + /// Writes the template string and params arguments to the log with the specified priority log level + /// </summary> + /// <param name="level">The log priority level</param> + /// <param name="value">The log template string</param> + /// <param name="args">Variable length array of objects to log with the specified templatre</param> + void Write(LogLevel level, string value, params ValueType[] args); + + /// <summary> + /// Gets the underlying log source + /// </summary> + /// <returns>The underlying log source</returns> + object GetLogProvider(); + /// <summary> + /// Gets the underlying log source + /// </summary> + /// <returns>The underlying log source</returns> + public virtual T GetLogProvider<T>() => (T)GetLogProvider(); + } +} diff --git a/Utils/src/Logging/LogLevel.cs b/Utils/src/Logging/LogLevel.cs new file mode 100644 index 0000000..1851c26 --- /dev/null +++ b/Utils/src/Logging/LogLevel.cs @@ -0,0 +1,33 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: LogLevel.cs +* +* LogLevel.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Logging +{ + public enum LogLevel + { + Verbose, Debug, Information, Warning, Error, Fatal + } +} diff --git a/Utils/src/Logging/LoggerExtensions.cs b/Utils/src/Logging/LoggerExtensions.cs new file mode 100644 index 0000000..cd314ed --- /dev/null +++ b/Utils/src/Logging/LoggerExtensions.cs @@ -0,0 +1,60 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: LoggerExtensions.cs +* +* LoggerExtensions.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +#pragma warning disable CA1062 // Validate arguments of public methods + +using System; + +namespace VNLib.Utils.Logging +{ + /// <summary> + /// Extension helper methods for writing logs to a <see cref="ILogProvider"/> + /// </summary> + public static class LoggerExtensions + { + public static void Debug(this ILogProvider log, Exception exp, string value = "") => log.Write(LogLevel.Debug, exp, value); + public static void Debug(this ILogProvider log, string value) => log.Write(LogLevel.Debug, value); + public static void Debug(this ILogProvider log, string format, params object?[] args) => log.Write(LogLevel.Debug, format, args); + public static void Debug(this ILogProvider log, string format, params ValueType[] args) => log.Write(LogLevel.Debug, format, args); + public static void Error(this ILogProvider log, Exception exp, string value = "") => log.Write(LogLevel.Error, exp, value); + public static void Error(this ILogProvider log, string value) => log.Write(LogLevel.Error, value); + public static void Error(this ILogProvider log, string format, params object?[] args) => log.Write(LogLevel.Error, format, args); + public static void Fatal(this ILogProvider log, Exception exp, string value = "") => log.Write(LogLevel.Fatal, exp, value); + public static void Fatal(this ILogProvider log, string value) => log.Write(LogLevel.Fatal, value); + public static void Fatal(this ILogProvider log, string format, params object?[] args) => log.Write(LogLevel.Fatal, format, args); + public static void Fatal(this ILogProvider log, string format, params ValueType[] args) => log.Write(LogLevel.Fatal, format, args); + public static void Information(this ILogProvider log, Exception exp, string value = "") => log.Write(LogLevel.Information, exp, value); + public static void Information(this ILogProvider log, string value) => log.Write(LogLevel.Information, value); + public static void Information(this ILogProvider log, string format, params object?[] args) => log.Write(LogLevel.Information, format, args); + public static void Information(this ILogProvider log, string format, params ValueType[] args) => log.Write(LogLevel.Information, format, args); + public static void Verbose(this ILogProvider log, Exception exp, string value = "") => log.Write(LogLevel.Verbose, exp, value); + public static void Verbose(this ILogProvider log, string value) => log.Write(LogLevel.Verbose, value); + public static void Verbose(this ILogProvider log, string format, params object?[] args) => log.Write(LogLevel.Verbose, format, args); + public static void Verbose(this ILogProvider log, string format, params ValueType[] args) => log.Write(LogLevel.Verbose, format, args); + public static void Warn(this ILogProvider log, Exception exp, string value = "") => log.Write(LogLevel.Warning, exp, value); + public static void Warn(this ILogProvider log, string value) => log.Write(LogLevel.Warning, value); + public static void Warn(this ILogProvider log, string format, params object?[] args) => log.Write(LogLevel.Warning, format, args); + public static void Warn(this ILogProvider log, string format, params ValueType[] args) => log.Write(LogLevel.Warning, format, args); + } +} diff --git a/Utils/src/Memory/Caching/ICacheHolder.cs b/Utils/src/Memory/Caching/ICacheHolder.cs new file mode 100644 index 0000000..19eee64 --- /dev/null +++ b/Utils/src/Memory/Caching/ICacheHolder.cs @@ -0,0 +1,45 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ICacheHolder.cs +* +* ICacheHolder.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// Exposes basic control of classes that manage private caches + /// </summary> + public interface ICacheHolder + { + /// <summary> + /// Clears all held caches without causing application stopping effects. + /// </summary> + /// <remarks>This is a safe "light" cache clear</remarks> + void CacheClear(); + /// <summary> + /// Performs all necessary actions to clear all held caches immediatly. + /// </summary> + /// <remarks>A "hard" cache clear/reset regardless of cost</remarks> + void CacheHardClear(); + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/Caching/ICacheable.cs b/Utils/src/Memory/Caching/ICacheable.cs new file mode 100644 index 0000000..37575cc --- /dev/null +++ b/Utils/src/Memory/Caching/ICacheable.cs @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ICacheable.cs +* +* ICacheable.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// Represents a cacheable entity with an expiration + /// </summary> + public interface ICacheable : IEquatable<ICacheable> + { + /// <summary> + /// A <see cref="DateTime"/> value that the entry is no longer valid + /// </summary> + DateTime Expires { get; set; } + + /// <summary> + /// Invoked when a collection occurs + /// </summary> + void Evicted(); + } +} diff --git a/Utils/src/Memory/Caching/IObjectRental.cs b/Utils/src/Memory/Caching/IObjectRental.cs new file mode 100644 index 0000000..d9489f4 --- /dev/null +++ b/Utils/src/Memory/Caching/IObjectRental.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IObjectRental.cs +* +* IObjectRental.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Utils.Memory.Caching +{ + + /// <summary> + /// A thread safe store for reusing CLR managed objects + /// </summary> + /// <typeparam name="T">The reusable object class</typeparam> + public interface IObjectRental<T> where T: class + { + /// <summary> + /// Gets an object from the store, or creates a new one if none are available + /// </summary> + /// <returns>An instance of <typeparamref name="T"/> from the store if available or a new instance if none were available</returns> + T Rent(); + + /// <summary> + /// Returns a rented object back to the rental store for reuse + /// </summary> + /// <param name="item">The previously rented item</param> + void Return(T item); + } + +}
\ No newline at end of file diff --git a/Utils/src/Memory/Caching/IReusable.cs b/Utils/src/Memory/Caching/IReusable.cs new file mode 100644 index 0000000..618878f --- /dev/null +++ b/Utils/src/Memory/Caching/IReusable.cs @@ -0,0 +1,42 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IReusable.cs +* +* IReusable.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// Allows for use within a <see cref="ReusableStore{T}"/>, this object is intended to be reused heavily + /// </summary> + public interface IReusable + { + /// <summary> + /// The instance should prepare itself for use (or re-use) + /// </summary> + void Prepare(); + /// <summary> + /// The intance is being returned and should determine if it's state is reusabled + /// </summary> + /// <returns>true if the instance can/should be reused, false if it should not be reused</returns> + bool Release(); + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/Caching/LRUCache.cs b/Utils/src/Memory/Caching/LRUCache.cs new file mode 100644 index 0000000..7e96e0a --- /dev/null +++ b/Utils/src/Memory/Caching/LRUCache.cs @@ -0,0 +1,127 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: LRUCache.cs +* +* LRUCache.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// A base class for a Least Recently Used cache + /// </summary> + /// <typeparam name="TKey">The key for O(1) lookups</typeparam> + /// <typeparam name="TValue">The value to store within cache</typeparam> + public abstract class LRUCache<TKey, TValue> : LRUDataStore<TKey, TValue> where TKey : notnull + { + ///<inheritdoc/> + protected LRUCache() + {} + ///<inheritdoc/> + protected LRUCache(int initialCapacity) : base(initialCapacity) + {} + ///<inheritdoc/> + protected LRUCache(IEqualityComparer<TKey> keyComparer) : base(keyComparer) + {} + ///<inheritdoc/> + protected LRUCache(int initialCapacity, IEqualityComparer<TKey> keyComparer) : base(initialCapacity, keyComparer) + {} + + /// <summary> + /// The maximum number of items to store in LRU cache + /// </summary> + protected abstract int MaxCapacity { get; } + + /// <summary> + /// Adds a new record to the LRU cache + /// </summary> + /// <param name="item">A <see cref="KeyValuePair{TKey, TValue}"/> to add to the cache store</param> + public override void Add(KeyValuePair<TKey, TValue> item) + { + //See if the store is at max capacity and an item needs to be evicted + if(Count == MaxCapacity) + { + //A record needs to be evicted before a new record can be added + + //Get the oldest node from the list to reuse its instance and remove the old value + LinkedListNode<KeyValuePair<TKey, TValue>> oldNode = List.First!; //not null because count is at max capacity so an item must be at the end of the list + //Store old node value field + KeyValuePair<TKey, TValue> oldRecord = oldNode.Value; + //Remove from lookup + LookupTable.Remove(oldRecord.Key); + //Remove the node + List.RemoveFirst(); + //Reuse the old ll node + oldNode.Value = item; + //add lookup with new key + LookupTable.Add(item.Key, oldNode); + //Add to end of list + List.AddLast(oldNode); + //Invoke evicted method + Evicted(oldRecord); + } + else + { + //Add new item to the list + base.Add(item); + } + } + /// <summary> + /// Attempts to get a value by the given key. + /// </summary> + /// <param name="key">The key identifying the value to store</param> + /// <param name="value">The value to store</param> + /// <returns>A value indicating if the value was found in the store</returns> + public override bool TryGetValue(TKey key, [NotNullWhen(true)] out TValue? value) + { + //See if the cache contains the value + if(base.TryGetValue(key, out value)) + { + //Cache hit + return true; + } + //Cache miss + if(CacheMiss(key, out value)) + { + //Lookup hit + //Add the record to the store (eviction will happen as necessary + Add(key, value); + return true; + } + //Record does not exist + return false; + } + /// <summary> + /// Invoked when a record is evicted from the cache + /// </summary> + /// <param name="evicted">The record that is being evicted</param> + protected abstract void Evicted(KeyValuePair<TKey, TValue> evicted); + /// <summary> + /// Invoked when an entry was requested and was not found in cache. + /// </summary> + /// <param name="key">The key identifying the record to lookup</param> + /// <param name="value">The found value matching the key</param> + /// <returns>A value indicating if the record was found</returns> + protected abstract bool CacheMiss(TKey key, [NotNullWhen(true)] out TValue? value); + } +} diff --git a/Utils/src/Memory/Caching/LRUDataStore.cs b/Utils/src/Memory/Caching/LRUDataStore.cs new file mode 100644 index 0000000..f564fcc --- /dev/null +++ b/Utils/src/Memory/Caching/LRUDataStore.cs @@ -0,0 +1,232 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: LRUDataStore.cs +* +* LRUDataStore.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Linq; +using System.Collections; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// A Least Recently Used store base class for E2E O(1) operations + /// </summary> + /// <typeparam name="TKey">A key used for O(1) lookups</typeparam> + /// <typeparam name="TValue">A value to store</typeparam> + public abstract class LRUDataStore<TKey, TValue> : IDictionary<TKey, TValue>, IReadOnlyDictionary<TKey, TValue>, IReadOnlyCollection<TValue>, IEnumerable<KeyValuePair<TKey, TValue>> + where TKey: notnull + { + /// <summary> + /// A lookup table that provides O(1) access times for key-value lookups + /// </summary> + protected Dictionary<TKey, LinkedListNode<KeyValuePair<TKey, TValue>>> LookupTable { get; } + /// <summary> + /// A linked list that tracks the least recently used item. + /// New items (or recently access items) are moved to the end of the list. + /// The head contains the least recently used item + /// </summary> + protected LinkedList<KeyValuePair<TKey, TValue>> List { get; } + + /// <summary> + /// Initializes an empty <see cref="LRUDataStore{TKey, TValue}"/> + /// </summary> + protected LRUDataStore() + { + LookupTable = new(); + List = new(); + } + /// <summary> + /// Initializes an empty <see cref="LRUDataStore{TKey, TValue}"/> and sets + /// the lookup table's inital capacity + /// </summary> + /// <param name="initialCapacity">LookupTable initial capacity</param> + protected LRUDataStore(int initialCapacity) + { + LookupTable = new(initialCapacity); + List = new(); + } + /// <summary> + /// Initializes an empty <see cref="LRUDataStore{TKey, TValue}"/> and uses the + /// specified keycomparison + /// </summary> + /// <param name="keyComparer">A <see cref="IEqualityComparer{T}"/> used by the Lookuptable to compare keys</param> + protected LRUDataStore(IEqualityComparer<TKey> keyComparer) + { + LookupTable = new(keyComparer); + List = new(); + } + /// <summary> + /// Initializes an empty <see cref="LRUDataStore{TKey, TValue}"/> and uses the + /// specified keycomparison, and sets the lookup table's initial capacity + /// </summary> + /// <param name="initialCapacity">LookupTable initial capacity</param> + /// <param name="keyComparer">A <see cref="IEqualityComparer{T}"/> used by the Lookuptable to compare keys</param> + protected LRUDataStore(int initialCapacity, IEqualityComparer<TKey> keyComparer) + { + LookupTable = new(initialCapacity, keyComparer); + List = new(); + } + + /// <summary> + /// Gets or sets a value within the LRU cache. + /// </summary> + /// <param name="key">The key identifying the value</param> + /// <returns>The value stored at the given key</returns> + /// <remarks>Items are promoted in the store when accessed</remarks> + public virtual TValue this[TKey key] + { + get + { + return TryGetValue(key, out TValue? value) + ? value + : throw new KeyNotFoundException("The item or its key were not found in the LRU data store"); + } + set + { + //If a node by the same key in the store exists, just replace its value + if(LookupTable.TryGetValue(key, out LinkedListNode<KeyValuePair<TKey, TValue>>? oldNode)) + { + //Remove the node before re-adding it + List.Remove(oldNode); + oldNode.Value = new KeyValuePair<TKey, TValue>(key, value); + //Move the item to the front of the list + List.AddLast(oldNode); + } + else + { + //Node does not exist yet so create new one + Add(key, value); + } + } + } + ///<inheritdoc/> + public ICollection<TKey> Keys => LookupTable.Keys; + ///<inheritdoc/> + ///<exception cref="NotImplementedException"></exception> + public ICollection<TValue> Values => throw new NotImplementedException(); + IEnumerable<TKey> IReadOnlyDictionary<TKey, TValue>.Keys => LookupTable.Keys; + IEnumerable<TValue> IReadOnlyDictionary<TKey, TValue>.Values => List.Select(static node => node.Value); + IEnumerator<TValue> IEnumerable<TValue>.GetEnumerator() => List.Select(static node => node.Value).GetEnumerator(); + + /// <summary> + /// Gets the number of items within the LRU store + /// </summary> + public int Count => List.Count; + ///<inheritdoc/> + public abstract bool IsReadOnly { get; } + + /// <summary> + /// Adds the specified record to the store and places it at the end of the LRU queue + /// </summary> + /// <param name="key">The key identifying the record</param> + /// <param name="value">The value to store at the key</param> + public void Add(TKey key, TValue value) + { + //Create new kvp lookup ref + KeyValuePair<TKey, TValue> lookupRef = new(key, value); + //Insert the lookup + Add(lookupRef); + } + ///<inheritdoc/> + public bool Remove(KeyValuePair<TKey, TValue> item) => Remove(item.Key); + ///<inheritdoc/> + IEnumerator IEnumerable.GetEnumerator() => List.GetEnumerator(); + ///<inheritdoc/> + public void CopyTo(KeyValuePair<TKey, TValue>[] array, int arrayIndex) => List.CopyTo(array, arrayIndex); + ///<inheritdoc/> + public virtual bool ContainsKey(TKey key) => LookupTable.ContainsKey(key); + ///<inheritdoc/> + public virtual IEnumerator<KeyValuePair<TKey, TValue>> GetEnumerator() => List.GetEnumerator(); + + /// <summary> + /// Adds the specified record to the store and places it at the end of the LRU queue + /// </summary> + /// <param name="item">The item to add</param> + public virtual void Add(KeyValuePair<TKey, TValue> item) + { + //Init new ll node + LinkedListNode<KeyValuePair<TKey, TValue>> newNode = new(item); + //Insert the new node + LookupTable.Add(item.Key, newNode); + //Add to the end of the linked list + List.AddLast(newNode); + } + /// <summary> + /// Removes all elements from the LRU store + /// </summary> + public virtual void Clear() + { + //Clear lists + LookupTable.Clear(); + List.Clear(); + } + /// <summary> + /// Determines if the <see cref="KeyValuePair{TKey, TValue}"/> exists in the store + /// </summary> + /// <param name="item">The record to search for</param> + /// <returns>True if the key was found in the store and the value equals the stored value, false otherwise</returns> + public virtual bool Contains(KeyValuePair<TKey, TValue> item) + { + if (LookupTable.TryGetValue(item.Key, out LinkedListNode<KeyValuePair<TKey, TValue>>? lookup)) + { + return lookup.Value.Value?.Equals(item.Value) ?? false; + } + return false; + } + ///<inheritdoc/> + public virtual bool Remove(TKey key) + { + //Remove the item from the lookup table and if it exists, remove the node from the list + if(LookupTable.Remove(key, out LinkedListNode<KeyValuePair<TKey, TValue>>? node)) + { + //Remove the new from the list + List.Remove(node); + return true; + } + return false; + } + /// <summary> + /// Tries to get a value from the store with its key. Found items are promoted + /// </summary> + /// <param name="key">The key identifying the value</param> + /// <param name="value">The found value</param> + /// <returns>A value indicating if the element was found in the store</returns> + public virtual bool TryGetValue(TKey key, [NotNullWhen(true)] out TValue? value) + { + //Lookup the + if (LookupTable.TryGetValue(key, out LinkedListNode<KeyValuePair<TKey, TValue>>? val)) + { + //Remove the value from the list and add it to the front of the list + List.Remove(val); + List.AddLast(val); + value = val.Value.Value!; + return true; + } + value = default; + return false; + } + + } +} diff --git a/Utils/src/Memory/Caching/ObjectRental.cs b/Utils/src/Memory/Caching/ObjectRental.cs new file mode 100644 index 0000000..22aca95 --- /dev/null +++ b/Utils/src/Memory/Caching/ObjectRental.cs @@ -0,0 +1,236 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ObjectRental.cs +* +* ObjectRental.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Diagnostics; +using System.Collections; +using System.Collections.Generic; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory.Caching +{ + //TODO: implement lock-free object tracking + + /// <summary> + /// Provides concurrent storage for reusable objects to be rented and returned. This class + /// and its members is thread-safe + /// </summary> + /// <typeparam name="T">The data type to reuse</typeparam> + public class ObjectRental<T> : ObjectRental, IObjectRental<T>, ICacheHolder, IEnumerable<T> where T: class + { + /// <summary> + /// The initial data-structure capacity if quota is not defined + /// </summary> + public const int INITIAL_STRUCTURE_SIZE = 50; + + protected readonly SemaphoreSlim StorageLock; + protected readonly Stack<T> Storage; + protected readonly HashSet<T> ContainsStore; + + protected readonly Action<T>? ReturnAction; + protected readonly Action<T>? RentAction; + protected readonly Func<T> Constructor; + /// <summary> + /// Is the object type in the current store implement the Idisposable interface? + /// </summary> + protected readonly bool IsDisposableType; + + /// <summary> + /// The maximum number of objects that will be cached. + /// Once this threshold has been reached, objects are + /// no longer stored + /// </summary> + protected readonly int QuotaLimit; + +#pragma warning disable CS8618 //Internal constructor does not set the constructor function + private ObjectRental(int quota) +#pragma warning restore CS8618 + { + //alloc new stack for rentals + Storage = new(Math.Max(quota, INITIAL_STRUCTURE_SIZE)); + //Hashtable for quick lookups + ContainsStore = new(Math.Max(quota, INITIAL_STRUCTURE_SIZE)); + //Semaphore slim to provide exclusive access + StorageLock = new SemaphoreSlim(1, 1); + //Store quota, if quota is -1, set to int-max to "disable quota" + QuotaLimit = quota == 0 ? int.MaxValue : quota; + //Determine if the type is disposeable and store a local value + IsDisposableType = typeof(IDisposable).IsAssignableFrom(typeof(T)); + } + + /// <summary> + /// Creates a new <see cref="ObjectRental{T}"/> store with the rent/return callback methods + /// </summary> + /// <param name="constructor">The type initializer</param> + /// <param name="rentCb">The pre-retnal preperation action</param> + /// <param name="returnCb">The pre-return cleanup action</param> + /// <param name="quota">The maximum number of elements to cache in the store</param> + protected internal ObjectRental(Func<T> constructor, Action<T>? rentCb, Action<T>? returnCb, int quota) : this(quota) + { + this.RentAction = rentCb; + this.ReturnAction = returnCb; + this.Constructor = constructor; + } + + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + public virtual T Rent() + { + Check(); + //See if we have an available object, if not return a new one by invoking the constructor function + T? rental = default; + //Get lock + using (SemSlimReleaser releader = StorageLock.GetReleaser()) + { + //See if the store contains an item ready to use + if(Storage.TryPop(out T? item)) + { + rental = item; + //Remove the item from the hash table + ContainsStore.Remove(item); + } + } + //If no object was removed from the store, create a new one + rental ??= Constructor(); + //If rental cb is defined, invoke it + RentAction?.Invoke(rental); + return rental; + } + + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + public virtual void Return(T item) + { + Check(); + //Invoke return callback if set + ReturnAction?.Invoke(item); + //Keeps track to know if the element was added or need to be cleaned up + bool wasAdded = false; + using (SemSlimReleaser releader = StorageLock.GetReleaser()) + { + //Check quota limit + if (Storage.Count < QuotaLimit) + { + //Store item if it doesnt exist already + if (ContainsStore.Add(item)) + { + //Store the object + Storage.Push(item); + } + //Set the was added flag + wasAdded = true; + } + } + if (!wasAdded && IsDisposableType) + { + //If the element was not added and is disposeable, we can dispose the element + (item as IDisposable)!.Dispose(); + //Write debug message + Debug.WriteLine("Object rental disposed an object over quota"); + } + } + + /// <remarks> + /// NOTE: If <typeparamref name="T"/> implements <see cref="IDisposable"/> + /// interface, this method does nothing + /// </remarks> + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + public virtual void CacheClear() + { + Check(); + //If the type is disposeable, cleaning can be a long process, so defer to hard clear + if (IsDisposableType) + { + return; + } + //take the semaphore + using SemSlimReleaser releader = StorageLock.GetReleaser(); + //Clear stores + ContainsStore.Clear(); + Storage.Clear(); + } + + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + public virtual void CacheHardClear() + { + Check(); + //take the semaphore + using SemSlimReleaser releader = StorageLock.GetReleaser(); + //If the type is disposable, dispose all elements before clearing storage + if (IsDisposableType) + { + //Dispose all elements + foreach (T element in Storage.ToArray()) + { + (element as IDisposable)!.Dispose(); + } + } + //Clear the storeage + Storage.Clear(); + ContainsStore.Clear(); + } + ///<inheritdoc/> + protected override void Free() + { + StorageLock.Dispose(); + //If the element type is disposable, dispose all elements on a hard clear + if (IsDisposableType) + { + //Get all elements + foreach (T element in Storage.ToArray()) + { + (element as IDisposable)!.Dispose(); + } + } + } + + ///<inheritdoc/> + public IEnumerator<T> GetEnumerator() + { + Check(); + //Enter the semaphore + using SemSlimReleaser releader = StorageLock.GetReleaser(); + foreach (T item in Storage) + { + yield return item; + } + } + + IEnumerator IEnumerable.GetEnumerator() + { + Check(); + //Enter the semaphore + using SemSlimReleaser releader = StorageLock.GetReleaser(); + foreach (T item in Storage) + { + yield return item; + } + } + } + +}
\ No newline at end of file diff --git a/Utils/src/Memory/Caching/ObjectRentalBase.cs b/Utils/src/Memory/Caching/ObjectRentalBase.cs new file mode 100644 index 0000000..305d93f --- /dev/null +++ b/Utils/src/Memory/Caching/ObjectRentalBase.cs @@ -0,0 +1,155 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ObjectRentalBase.cs +* +* ObjectRentalBase.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// Provides concurrent storage for reusable objects to be rented and returned. This class + /// and its members is thread-safe + /// </summary> + public abstract class ObjectRental : VnDisposeable + { + /// <summary> + /// Creates a new <see cref="ObjectRental{T}"/> store + /// </summary> + /// <param name="quota">The maximum number of elements that will be cached</param> + public static ObjectRental<TNew> Create<TNew>(int quota = 0) where TNew : class, new() + { + static TNew constructor() => new(); + return new ObjectRental<TNew>(constructor, null, null, quota); + } + /// <summary> + /// Creates a new <see cref="ObjectRental{T}"/> store with generic rental and return callback handlers + /// </summary> + /// <param name="rentCb">Function responsible for preparing an instance to be rented</param> + /// <param name="returnCb">Function responsible for cleaning up an instance before reuse</param> + /// <param name="quota">The maximum number of elements that will be cached</param> + public static ObjectRental<TNew> Create<TNew>(Action<TNew>? rentCb, Action<TNew>? returnCb, int quota = 0) where TNew : class, new() + { + static TNew constructor() => new(); + return new ObjectRental<TNew>(constructor, rentCb, returnCb, quota); + } + /// <summary> + /// Creates a new <see cref="ObjectRental{T}"/> store with a generic constructor function + /// </summary> + /// <param name="constructor">The function invoked to create a new instance when required</param> + /// <param name="quota">The maximum number of elements that will be cached</param> + /// <returns></returns> + public static ObjectRental<TNew> Create<TNew>(Func<TNew> constructor, int quota = 0) where TNew: class + { + return new ObjectRental<TNew>(constructor, null, null, quota); + } + /// <summary> + /// Creates a new <see cref="ObjectRental{T}"/> store with generic rental and return callback handlers + /// </summary> + /// <param name="constructor">The function invoked to create a new instance when required</param> + /// <param name="rentCb">Function responsible for preparing an instance to be rented</param> + /// <param name="returnCb">Function responsible for cleaning up an instance before reuse</param> + /// <param name="quota">The maximum number of elements that will be cached</param> + public static ObjectRental<TNew> Create<TNew>(Func<TNew> constructor, Action<TNew>? rentCb, Action<TNew>? returnCb, int quota = 0) where TNew : class + { + return new ObjectRental<TNew>(constructor, rentCb, returnCb, quota); + } + + /// <summary> + /// Creates a new <see cref="ThreadLocalObjectStorage{TNew}"/> store with generic rental and return callback handlers + /// </summary> + /// <typeparam name="TNew"></typeparam> + /// <param name="constructor">The function invoked to create a new instance when required</param> + /// <param name="rentCb">Function responsible for preparing an instance to be rented</param> + /// <param name="returnCb">Function responsible for cleaning up an instance before reuse</param> + /// <returns>The initialized store</returns> + public static ThreadLocalObjectStorage<TNew> CreateThreadLocal<TNew>(Func<TNew> constructor, Action<TNew>? rentCb, Action<TNew>? returnCb) where TNew : class + { + return new ThreadLocalObjectStorage<TNew>(constructor, rentCb, returnCb); + } + /// <summary> + /// Creates a new <see cref="ThreadLocalObjectStorage{T}"/> store with generic rental and return callback handlers + /// </summary> + /// <param name="rentCb">Function responsible for preparing an instance to be rented</param> + /// <param name="returnCb">Function responsible for cleaning up an instance before reuse</param> + public static ThreadLocalObjectStorage<TNew> CreateThreadLocal<TNew>(Action<TNew>? rentCb, Action<TNew>? returnCb) where TNew : class, new() + { + static TNew constructor() => new(); + return new ThreadLocalObjectStorage<TNew>(constructor, rentCb, returnCb); + } + /// <summary> + /// Creates a new <see cref="ThreadLocalObjectStorage{T}"/> store + /// </summary> + public static ThreadLocalObjectStorage<TNew> CreateThreadLocal<TNew>() where TNew : class, new() + { + static TNew constructor() => new(); + return new ThreadLocalObjectStorage<TNew>(constructor, null, null); + } + /// <summary> + /// Creates a new <see cref="ThreadLocalObjectStorage{T}"/> store with a generic constructor function + /// </summary> + /// <param name="constructor">The function invoked to create a new instance when required</param> + /// <returns></returns> + public static ThreadLocalObjectStorage<TNew> CreateThreadLocal<TNew>(Func<TNew> constructor) where TNew : class + { + return new ThreadLocalObjectStorage<TNew>(constructor, null, null); + } + + /// <summary> + /// Creates a new <see cref="ReusableStore{T}"/> instance with a parameterless constructor + /// </summary> + /// <typeparam name="T">The <see cref="IReusable"/> type</typeparam> + /// <param name="quota">The maximum number of elements that will be cached</param> + /// <returns></returns> + public static ReusableStore<T> CreateReusable<T>(int quota = 0) where T : class, IReusable, new() + { + static T constructor() => new(); + return new(constructor, quota); + } + /// <summary> + /// Creates a new <see cref="ReusableStore{T}"/> instance with the specified constructor + /// </summary> + /// <typeparam name="T">The <see cref="IReusable"/> type</typeparam> + /// <param name="constructor">The constructor function invoked to create new instances of the <see cref="IReusable"/> type</param> + /// <param name="quota">The maximum number of elements that will be cached</param> + /// <returns></returns> + public static ReusableStore<T> CreateReusable<T>(Func<T> constructor, int quota = 0) where T : class, IReusable => new(constructor, quota); + + /// <summary> + /// Creates a new <see cref="ThreadLocalReusableStore{T}"/> instance with a parameterless constructor + /// </summary> + /// <typeparam name="T">The <see cref="IReusable"/> type</typeparam> + /// <returns></returns> + public static ThreadLocalReusableStore<T> CreateThreadLocalReusable<T>() where T : class, IReusable, new() + { + static T constructor() => new(); + return new(constructor); + } + /// <summary> + /// Creates a new <see cref="ThreadLocalReusableStore{T}"/> instance with the specified constructor + /// </summary> + /// <typeparam name="T">The <see cref="IReusable"/> type</typeparam> + /// <param name="constructor">The constructor function invoked to create new instances of the <see cref="IReusable"/> type</param> + /// <returns></returns> + public static ThreadLocalReusableStore<T> CreateThreadLocalReusable<T>(Func<T> constructor) where T : class, IReusable => new(constructor); + } +} diff --git a/Utils/src/Memory/Caching/ReusableStore.cs b/Utils/src/Memory/Caching/ReusableStore.cs new file mode 100644 index 0000000..aacd012 --- /dev/null +++ b/Utils/src/Memory/Caching/ReusableStore.cs @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ReusableStore.cs +* +* ReusableStore.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory.Caching +{ + + /// <summary> + /// A reusable object store that extends <see cref="ObjectRental{T}"/>, that allows for objects to be reused heavily + /// </summary> + /// <typeparam name="T">A reusable object</typeparam> + public class ReusableStore<T> : ObjectRental<T> where T : class, IReusable + { + internal ReusableStore(Func<T> constructor, int quota) :base(constructor, null, null, quota) + {} + ///<inheritdoc/> + public override T Rent() + { + //Rent the object (or create it) + T rental = base.Rent(); + //Invoke prepare function + rental.Prepare(); + //return object + return rental; + } + ///<inheritdoc/> + public override void Return(T item) + { + /* + * Clean up the item by invoking the cleanup function, + * and only return the item for reuse if the caller allows + */ + if (item.Release()) + { + base.Return(item); + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/Caching/ThreadLocalObjectStorage.cs b/Utils/src/Memory/Caching/ThreadLocalObjectStorage.cs new file mode 100644 index 0000000..511af24 --- /dev/null +++ b/Utils/src/Memory/Caching/ThreadLocalObjectStorage.cs @@ -0,0 +1,76 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ThreadLocalObjectStorage.cs +* +* ThreadLocalObjectStorage.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// Derrives from <see cref="ObjectRental{T}"/> to provide object rental syntax for <see cref="ThreadLocal{T}"/> + /// storage + /// </summary> + /// <typeparam name="T">The data type to store</typeparam> + public class ThreadLocalObjectStorage<T> : ObjectRental<T> where T: class + { + protected ThreadLocal<T> Store { get; } + + internal ThreadLocalObjectStorage(Func<T> constructor, Action<T>? rentCb, Action<T>? returnCb) + :base(constructor, rentCb, returnCb, 0) + { + Store = new(Constructor); + } + + /// <summary> + /// "Rents" or creates an object for the current thread + /// </summary> + /// <returns>The new or stored instanced</returns> + /// <exception cref="ObjectDisposedException"></exception> + public override T Rent() + { + Check(); + //Get the tlocal value + T value = Store.Value!; + //Invoke the rent action if set + base.RentAction?.Invoke(value); + return value; + } + + /// <inheritdoc/> + /// <exception cref="ObjectDisposedException"></exception> + public override void Return(T item) + { + Check(); + //Invoke the rent action + base.ReturnAction?.Invoke(item); + } + + ///<inheritdoc/> + protected override void Free() + { + Store.Dispose(); + base.Free(); + } + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/Caching/ThreadLocalReusableStore.cs b/Utils/src/Memory/Caching/ThreadLocalReusableStore.cs new file mode 100644 index 0000000..83cd4d6 --- /dev/null +++ b/Utils/src/Memory/Caching/ThreadLocalReusableStore.cs @@ -0,0 +1,64 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ThreadLocalReusableStore.cs +* +* ThreadLocalReusableStore.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory.Caching +{ + /// <summary> + /// A reusable object store that extends <see cref="ThreadLocalObjectStorage{T}"/>, that allows for objects to be reused heavily + /// in a thread-local cache + /// </summary> + /// <typeparam name="T">A reusable object</typeparam> + public class ThreadLocalReusableStore<T> : ThreadLocalObjectStorage<T> where T: class, IReusable + { + /// <summary> + /// Creates a new <see cref="ThreadLocalReusableStore{T}"/> instance + /// </summary> + internal ThreadLocalReusableStore(Func<T> constructor):base(constructor, null, null) + { } + ///<inheritdoc/> + public override T Rent() + { + //Rent the object (or create it) + T rental = base.Rent(); + //Invoke prepare function + rental.Prepare(); + //return object + return rental; + } + ///<inheritdoc/> + public override void Return(T item) + { + /* + * Clean up the item by invoking the cleanup function, + * and only return the item for reuse if the caller allows + */ + if (item.Release()) + { + base.Return(item); + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/ForwardOnlyBufferWriter.cs b/Utils/src/Memory/ForwardOnlyBufferWriter.cs new file mode 100644 index 0000000..0ea507e --- /dev/null +++ b/Utils/src/Memory/ForwardOnlyBufferWriter.cs @@ -0,0 +1,122 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ForwardOnlyBufferWriter.cs +* +* ForwardOnlyBufferWriter.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a stack based buffer writer + /// </summary> + public ref struct ForwardOnlyWriter<T> + { + /// <summary> + /// The buffer for writing output data to + /// </summary> + public readonly Span<T> Buffer { get; } + /// <summary> + /// The number of characters written to the buffer + /// </summary> + public int Written { readonly get; set; } + /// <summary> + /// The number of characters remaining in the buffer + /// </summary> + public readonly int RemainingSize => Buffer.Length - Written; + + /// <summary> + /// The remaining buffer window + /// </summary> + public readonly Span<T> Remaining => Buffer[Written..]; + + /// <summary> + /// Creates a new <see cref="ForwardOnlyWriter{T}"/> assigning the specified buffer + /// </summary> + /// <param name="buffer">The buffer to write data to</param> + public ForwardOnlyWriter(in Span<T> buffer) + { + Buffer = buffer; + Written = 0; + } + + /// <summary> + /// Returns a compiled string from the characters written to the buffer + /// </summary> + /// <returns>A string of the characters written to the buffer</returns> + public readonly override string ToString() => Buffer[..Written].ToString(); + + /// <summary> + /// Appends a sequence to the buffer + /// </summary> + /// <param name="data">The data to append to the buffer</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Append(ReadOnlySpan<T> data) + { + //Make sure the current window is large enough to buffer the new string + if (data.Length > RemainingSize) + { + throw new ArgumentOutOfRangeException(nameof(Remaining) ,"The internal buffer does not have enough buffer space"); + } + Span<T> window = Buffer[Written..]; + //write data to window + data.CopyTo(window); + //update char position + Written += data.Length; + } + /// <summary> + /// Appends a single item to the buffer + /// </summary> + /// <param name="c">The item to append to the buffer</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Append(T c) + { + //Make sure the current window is large enough to buffer the new string + if (RemainingSize == 0) + { + throw new ArgumentOutOfRangeException(nameof(Remaining), "The internal buffer does not have enough buffer space"); + } + //Write data to buffer and increment the buffer position + Buffer[Written++] = c; + } + + /// <summary> + /// Advances the writer forward the specifed number of elements + /// </summary> + /// <param name="count">The number of elements to advance the writer by</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Advance(int count) + { + if (count > RemainingSize) + { + throw new ArgumentOutOfRangeException(nameof(Remaining), "The internal buffer does not have enough buffer space"); + } + Written += count; + } + + /// <summary> + /// Resets the writer by setting the <see cref="Written"/> + /// property to 0. + /// </summary> + public void Reset() => Written = 0; + } +} diff --git a/Utils/src/Memory/ForwardOnlyMemoryReader.cs b/Utils/src/Memory/ForwardOnlyMemoryReader.cs new file mode 100644 index 0000000..c850b14 --- /dev/null +++ b/Utils/src/Memory/ForwardOnlyMemoryReader.cs @@ -0,0 +1,74 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ForwardOnlyMemoryReader.cs +* +* ForwardOnlyMemoryReader.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// A mutable structure used to implement a simple foward only + /// reader for a memory segment + /// </summary> + /// <typeparam name="T">The element type</typeparam> + public struct ForwardOnlyMemoryReader<T> + { + private readonly ReadOnlyMemory<T> _segment; + private readonly int _size; + + private int _position; + + /// <summary> + /// Initializes a new <see cref="FordwardOnlyMemoryReader{T}"/> + /// of the specified type using the specified internal buffer + /// </summary> + /// <param name="buffer">The buffer to read from</param> + public ForwardOnlyMemoryReader(in ReadOnlyMemory<T> buffer) + { + _segment = buffer; + _size = buffer.Length; + _position = 0; + } + + /// <summary> + /// The remaining data window + /// </summary> + public readonly ReadOnlyMemory<T> Window => _segment[_position..]; + /// <summary> + /// The number of elements remaining in the window + /// </summary> + public readonly int WindowSize => _size - _position; + + + /// <summary> + /// Advances the window position the specified number of elements + /// </summary> + /// <param name="count">The number of elements to advance the widnow position</param> + public void Advance(int count) => _position += count; + + /// <summary> + /// Resets the sliding window to the begining of the buffer + /// </summary> + public void Reset() => _position = 0; + } +} diff --git a/Utils/src/Memory/ForwardOnlyMemoryWriter.cs b/Utils/src/Memory/ForwardOnlyMemoryWriter.cs new file mode 100644 index 0000000..4f5286d --- /dev/null +++ b/Utils/src/Memory/ForwardOnlyMemoryWriter.cs @@ -0,0 +1,122 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ForwardOnlyMemoryWriter.cs +* +* ForwardOnlyMemoryWriter.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a mutable sliding buffer writer + /// </summary> + public struct ForwardOnlyMemoryWriter<T> + { + /// <summary> + /// The buffer for writing output data to + /// </summary> + public readonly Memory<T> Buffer { get; } + /// <summary> + /// The number of characters written to the buffer + /// </summary> + public int Written { readonly get; set; } + /// <summary> + /// The number of characters remaining in the buffer + /// </summary> + public readonly int RemainingSize => Buffer.Length - Written; + + /// <summary> + /// The remaining buffer window + /// </summary> + public readonly Memory<T> Remaining => Buffer[Written..]; + + /// <summary> + /// Creates a new <see cref="ForwardOnlyWriter{T}"/> assigning the specified buffer + /// </summary> + /// <param name="buffer">The buffer to write data to</param> + public ForwardOnlyMemoryWriter(in Memory<T> buffer) + { + Buffer = buffer; + Written = 0; + } + + /// <summary> + /// Returns a compiled string from the characters written to the buffer + /// </summary> + /// <returns>A string of the characters written to the buffer</returns> + public readonly override string ToString() => Buffer[..Written].ToString(); + + /// <summary> + /// Appends a sequence to the buffer + /// </summary> + /// <param name="data">The data to append to the buffer</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Append(ReadOnlyMemory<T> data) + { + //Make sure the current window is large enough to buffer the new string + if (data.Length > RemainingSize) + { + throw new ArgumentOutOfRangeException(nameof(Remaining), "The internal buffer does not have enough buffer space"); + } + Memory<T> window = Buffer[Written..]; + //write data to window + data.CopyTo(window); + //update char position + Written += data.Length; + } + /// <summary> + /// Appends a single item to the buffer + /// </summary> + /// <param name="c">The item to append to the buffer</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Append(T c) + { + //Make sure the current window is large enough to buffer the new string + if (RemainingSize == 0) + { + throw new ArgumentOutOfRangeException(nameof(Remaining), "The internal buffer does not have enough buffer space"); + } + //Write data to buffer and increment the buffer position + Buffer.Span[Written++] = c; + } + + /// <summary> + /// Advances the writer forward the specifed number of elements + /// </summary> + /// <param name="count">The number of elements to advance the writer by</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Advance(int count) + { + if (count > RemainingSize) + { + throw new ArgumentOutOfRangeException(nameof(count), count, "Cannot advance past the end of the buffer"); + } + Written += count; + } + + /// <summary> + /// Resets the writer by setting the <see cref="Written"/> + /// property to 0. + /// </summary> + public void Reset() => Written = 0; + } +} diff --git a/Utils/src/Memory/ForwardOnlyReader.cs b/Utils/src/Memory/ForwardOnlyReader.cs new file mode 100644 index 0000000..aa268c4 --- /dev/null +++ b/Utils/src/Memory/ForwardOnlyReader.cs @@ -0,0 +1,74 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ForwardOnlyReader.cs +* +* ForwardOnlyReader.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// A mutable structure used to implement a simple foward only + /// reader for a memory segment + /// </summary> + /// <typeparam name="T">The element type</typeparam> + public ref struct ForwardOnlyReader<T> + { + private readonly ReadOnlySpan<T> _segment; + private readonly int _size; + + private int _position; + + /// <summary> + /// Initializes a new <see cref="FordwardOnlyReader{T}"/> + /// of the specified type using the specified internal buffer + /// </summary> + /// <param name="buffer">The buffer to read from</param> + public ForwardOnlyReader(in ReadOnlySpan<T> buffer) + { + _segment = buffer; + _size = buffer.Length; + _position = 0; + } + + /// <summary> + /// The remaining data window + /// </summary> + public readonly ReadOnlySpan <T> Window => _segment[_position..]; + + /// <summary> + /// The number of elements remaining in the window + /// </summary> + public readonly int WindowSize => _size - _position; + + /// <summary> + /// Advances the window position the specified number of elements + /// </summary> + /// <param name="count">The number of elements to advance the widnow position</param> + public void Advance(int count) => _position += count; + + /// <summary> + /// Resets the sliding window to the begining of the buffer + /// </summary> + public void Reset() => _position = 0; + } +} diff --git a/Utils/src/Memory/IMemoryHandle.cs b/Utils/src/Memory/IMemoryHandle.cs new file mode 100644 index 0000000..75d1cce --- /dev/null +++ b/Utils/src/Memory/IMemoryHandle.cs @@ -0,0 +1,53 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IMemoryHandle.cs +* +* IMemoryHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Represents a handle for safe access to memory managed/unamanged memory + /// </summary> + /// <typeparam name="T">The type this handle represents</typeparam> + public interface IMemoryHandle<T> : IDisposable, IPinnable + { + /// <summary> + /// The size of the block as an integer + /// </summary> + /// <exception cref="OverflowException"></exception> + int IntLength { get; } + + /// <summary> + /// The number of elements in the block + /// </summary> + ulong Length { get; } + + /// <summary> + /// Gets the internal block as a span + /// </summary> + Span<T> Span { get; } + } + +} diff --git a/Utils/src/Memory/IStringSerializeable.cs b/Utils/src/Memory/IStringSerializeable.cs new file mode 100644 index 0000000..12cfe52 --- /dev/null +++ b/Utils/src/Memory/IStringSerializeable.cs @@ -0,0 +1,55 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IStringSerializeable.cs +* +* IStringSerializeable.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// A interface that provides indempodent abstractions for compiling an instance + /// to its representitive string. + /// </summary> + public interface IStringSerializeable + { + /// <summary> + /// Compiles the current instance into its safe string representation + /// </summary> + /// <returns>A string of the desired representation of the current instance</returns> + string Compile(); + /// <summary> + /// Compiles the current instance into its safe string representation, and writes its + /// contents to the specified buffer writer + /// </summary> + /// <param name="writer">The ouput writer to write the serialized representation to</param> + /// <exception cref="OutOfMemoryException"></exception> + void Compile(ref ForwardOnlyWriter<char> writer); + /// <summary> + /// Compiles the current instance into its safe string representation, and writes its + /// contents to the specified buffer writer + /// </summary> + /// <param name="buffer">The buffer to write the serialized representation to</param> + /// <returns>The number of characters written to the buffer</returns> + ERRNO Compile(in Span<char> buffer); + } +} diff --git a/Utils/src/Memory/IUnmangedHeap.cs b/Utils/src/Memory/IUnmangedHeap.cs new file mode 100644 index 0000000..5d8f4bf --- /dev/null +++ b/Utils/src/Memory/IUnmangedHeap.cs @@ -0,0 +1,59 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IUnmangedHeap.cs +* +* IUnmangedHeap.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Abstraction for handling (allocating, resizing, and freeing) blocks of unmanaged memory from an unmanged heap + /// </summary> + public interface IUnmangedHeap : IDisposable + { + /// <summary> + /// Allocates a block of memory from the heap and returns a pointer to the new memory block + /// </summary> + /// <param name="size">The size (in bytes) of the element</param> + /// <param name="elements">The number of elements to allocate</param> + /// <param name="zero">An optional parameter to zero the block of memory</param> + /// <returns></returns> + IntPtr Alloc(UInt64 elements, UInt64 size, bool zero); + + /// <summary> + /// Resizes the allocated block of memory to the new size + /// </summary> + /// <param name="block">The block to resize</param> + /// <param name="elements">The new number of elements</param> + /// <param name="size">The size (in bytes) of the type</param> + /// <param name="zero">An optional parameter to zero the block of memory</param> + void Resize(ref IntPtr block, UInt64 elements, UInt64 size, bool zero); + + /// <summary> + /// Free's a previously allocated block of memory + /// </summary> + /// <param name="block">The memory to be freed</param> + /// <returns>A value indicating if the free operation succeeded</returns> + bool Free(ref IntPtr block); + } +} diff --git a/Utils/src/Memory/Memory.cs b/Utils/src/Memory/Memory.cs new file mode 100644 index 0000000..822d98c --- /dev/null +++ b/Utils/src/Memory/Memory.cs @@ -0,0 +1,456 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: Memory.cs +* +* Memory.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Buffers; +using System.Security; +using System.Threading; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides optimized cross-platform maanged/umanaged safe/unsafe memory operations + /// </summary> + [SecurityCritical] + [ComVisible(false)] + public unsafe static class Memory + { + public const string SHARED_HEAP_TYPE_ENV= "VNLIB_SHARED_HEAP_TYPE"; + public const string SHARED_HEAP_INTIAL_SIZE_ENV = "VNLIB_SHARED_HEAP_SIZE"; + + /// <summary> + /// Initial shared heap size (bytes) + /// </summary> + public const ulong SHARED_HEAP_INIT_SIZE = 20971520; + + public const int MAX_BUF_SIZE = 2097152; + public const int MIN_BUF_SIZE = 16000; + + /// <summary> + /// The maximum buffer size requested by <see cref="UnsafeAlloc{T}(int, bool)"/> + /// that will use the array pool before falling back to the <see cref="Shared"/>. + /// heap. + /// </summary> + public const int MAX_UNSAFE_POOL_SIZE = 500 * 1024; + + /// <summary> + /// Provides a shared heap instance for the process to allocate memory from. + /// </summary> + /// <remarks> + /// The backing heap + /// is determined by the OS type and process environment varibles. + /// </remarks> + public static IUnmangedHeap Shared => _sharedHeap.Value; + + private static readonly Lazy<IUnmangedHeap> _sharedHeap; + + static Memory() + { + _sharedHeap = new Lazy<IUnmangedHeap>(() => InitHeapInternal(true), LazyThreadSafetyMode.PublicationOnly); + //Cleanup the heap on process exit + AppDomain.CurrentDomain.DomainUnload += DomainUnloaded; + } + + private static void DomainUnloaded(object sender, EventArgs e) + { + //Dispose the heap if allocated + if (_sharedHeap.IsValueCreated) + { + _sharedHeap.Value.Dispose(); + } + } + + /// <summary> + /// Initializes a new <see cref="IUnmangedHeap"/> determined by compilation/runtime flags + /// and operating system type for the current proccess. + /// </summary> + /// <returns>An <see cref="IUnmangedHeap"/> for the current process</returns> + /// <exception cref="SystemException"></exception> + /// <exception cref="DllNotFoundException"></exception> + public static IUnmangedHeap InitializeNewHeapForProcess() => InitHeapInternal(false); + + private static IUnmangedHeap InitHeapInternal(bool isShared) + { + bool IsWindows = OperatingSystem.IsWindows(); + //Get environment varable + string heapType = Environment.GetEnvironmentVariable(SHARED_HEAP_TYPE_ENV); + //Get inital size + string sharedSize = Environment.GetEnvironmentVariable(SHARED_HEAP_INTIAL_SIZE_ENV); + //Try to parse the shared size from the env + if (!ulong.TryParse(sharedSize, out ulong defaultSize)) + { + defaultSize = SHARED_HEAP_INIT_SIZE; + } + //Gen the private heap from its type or default + switch (heapType) + { + case "win32": + if (!IsWindows) + { + throw new PlatformNotSupportedException("Win32 private heaps are not supported on non-windows platforms"); + } + return PrivateHeap.Create(defaultSize); + case "rpmalloc": + //If the shared heap is being allocated, then return a lock free global heap + return isShared ? RpMallocPrivateHeap.GlobalHeap : new RpMallocPrivateHeap(false); + default: + return IsWindows ? PrivateHeap.Create(defaultSize) : new ProcessHeap(); + } + } + + /// <summary> + /// Gets a value that indicates if the Rpmalloc native library is loaded + /// </summary> + public static bool IsRpMallocLoaded { get; } = Environment.GetEnvironmentVariable(SHARED_HEAP_TYPE_ENV) == "rpmalloc"; + + #region Zero + /// <summary> + /// Zeros a block of memory of umanged type. If Windows is detected at runtime, calls RtlSecureZeroMemory Win32 function + /// </summary> + /// <typeparam name="T">Unmanged datatype</typeparam> + /// <param name="block">Block of memory to be cleared</param> + [MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)] + public static void UnsafeZeroMemory<T>(ReadOnlySpan<T> block) where T : unmanaged + { + if (!block.IsEmpty) + { + checked + { + fixed (void* ptr = &MemoryMarshal.GetReference(block)) + { + //Calls memset + Unsafe.InitBlock(ptr, 0, (uint)(block.Length * sizeof(T))); + } + } + } + } + /// <summary> + /// Zeros a block of memory of umanged type. If Windows is detected at runtime, calls RtlSecureZeroMemory Win32 function + /// </summary> + /// <typeparam name="T">Unmanged datatype</typeparam> + /// <param name="block">Block of memory to be cleared</param> + [MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)] + public static void UnsafeZeroMemory<T>(ReadOnlyMemory<T> block) where T : unmanaged + { + if (!block.IsEmpty) + { + checked + { + //Pin memory and get pointer + using MemoryHandle handle = block.Pin(); + //Calls memset + Unsafe.InitBlock(handle.Pointer, 0, (uint)(block.Length * sizeof(T))); + } + } + } + + /// <summary> + /// Initializes a block of memory with zeros + /// </summary> + /// <typeparam name="T">The unmanaged</typeparam> + /// <param name="block">The block of memory to initialize</param> + public static void InitializeBlock<T>(Span<T> block) where T : unmanaged => UnsafeZeroMemory<T>(block); + /// <summary> + /// Initializes a block of memory with zeros + /// </summary> + /// <typeparam name="T">The unmanaged</typeparam> + /// <param name="block">The block of memory to initialize</param> + public static void InitializeBlock<T>(Memory<T> block) where T : unmanaged => UnsafeZeroMemory<T>(block); + + /// <summary> + /// Zeroes a block of memory pointing to the structure + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="block">The pointer to the allocated structure</param> + public static void ZeroStruct<T>(IntPtr block) + { + //get thes size of the structure + int size = Unsafe.SizeOf<T>(); + //Zero block + Unsafe.InitBlock(block.ToPointer(), 0, (uint)size); + } + /// <summary> + /// Zeroes a block of memory pointing to the structure + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="structPtr">The pointer to the allocated structure</param> + public static void ZeroStruct<T>(void* structPtr) where T: unmanaged + { + //get thes size of the structure + int size = Unsafe.SizeOf<T>(); + //Zero block + Unsafe.InitBlock(structPtr, 0, (uint)size); + } + /// <summary> + /// Zeroes a block of memory pointing to the structure + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="structPtr">The pointer to the allocated structure</param> + public static void ZeroStruct<T>(T* structPtr) where T : unmanaged + { + //get thes size of the structure + int size = Unsafe.SizeOf<T>(); + //Zero block + Unsafe.InitBlock(structPtr, 0, (uint)size); + } + + #endregion + + #region Copy + /// <summary> + /// Copies data from source memory to destination memory of an umanged data type + /// </summary> + /// <typeparam name="T">Unmanged type</typeparam> + /// <param name="source">Source data <see cref="ReadOnlySpan{T}"/></param> + /// <param name="dest">Destination <see cref="MemoryHandle{T}"/></param> + /// <param name="destOffset">Dest offset</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static void Copy<T>(ReadOnlySpan<T> source, MemoryHandle<T> dest, Int64 destOffset) where T : unmanaged + { + if (source.IsEmpty) + { + return; + } + if (dest.Length < (ulong)(destOffset + source.Length)) + { + throw new ArgumentException("Source data is larger than the dest data block", nameof(source)); + } + //Get long offset from the destination handle + T* offset = dest.GetOffset(destOffset); + fixed(void* src = &MemoryMarshal.GetReference(source)) + { + int byteCount = checked(source.Length * sizeof(T)); + Unsafe.CopyBlock(offset, src, (uint)byteCount); + } + } + /// <summary> + /// Copies data from source memory to destination memory of an umanged data type + /// </summary> + /// <typeparam name="T">Unmanged type</typeparam> + /// <param name="source">Source data <see cref="ReadOnlyMemory{T}"/></param> + /// <param name="dest">Destination <see cref="MemoryHandle{T}"/></param> + /// <param name="destOffset">Dest offset</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static void Copy<T>(ReadOnlyMemory<T> source, MemoryHandle<T> dest, Int64 destOffset) where T : unmanaged + { + if (source.IsEmpty) + { + return; + } + if (dest.Length < (ulong)(destOffset + source.Length)) + { + throw new ArgumentException("Dest constraints are larger than the dest data block", nameof(source)); + } + //Get long offset from the destination handle + T* offset = dest.GetOffset(destOffset); + //Pin the source memory + using MemoryHandle srcHandle = source.Pin(); + int byteCount = checked(source.Length * sizeof(T)); + //Copy block using unsafe class + Unsafe.CopyBlock(offset, srcHandle.Pointer, (uint)byteCount); + } + /// <summary> + /// Copies data from source memory to destination memory of an umanged data type + /// </summary> + /// <typeparam name="T">Unmanged type</typeparam> + /// <param name="source">Source data <see cref="MemoryHandle{T}"/></param> + /// <param name="sourceOffset">Number of elements to offset source data</param> + /// <param name="dest">Destination <see cref="Span{T}"/></param> + /// <param name="destOffset">Dest offset</param> + /// <param name="count">Number of elements to copy</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static void Copy<T>(MemoryHandle<T> source, Int64 sourceOffset, Span<T> dest, int destOffset, int count) where T : unmanaged + { + if (count <= 0) + { + return; + } + if (source.Length < (ulong)(sourceOffset + count)) + { + throw new ArgumentException("Source constraints are larger than the source data block", nameof(count)); + } + if (dest.Length < destOffset + count) + { + throw new ArgumentOutOfRangeException(nameof(destOffset), "Destination offset range cannot exceed the size of the destination buffer"); + } + //Get offset to allow large blocks of memory + T* src = source.GetOffset(sourceOffset); + fixed(T* dst = &MemoryMarshal.GetReference(dest)) + { + //Cacl offset + T* dstoffset = dst + destOffset; + int byteCount = checked(count * sizeof(T)); + //Aligned copy + Unsafe.CopyBlock(dstoffset, src, (uint)byteCount); + } + } + /// <summary> + /// Copies data from source memory to destination memory of an umanged data type + /// </summary> + /// <typeparam name="T">Unmanged type</typeparam> + /// <param name="source">Source data <see cref="MemoryHandle{T}"/></param> + /// <param name="sourceOffset">Number of elements to offset source data</param> + /// <param name="dest">Destination <see cref="Memory{T}"/></param> + /// <param name="destOffset">Dest offset</param> + /// <param name="count">Number of elements to copy</param> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static void Copy<T>(MemoryHandle<T> source, Int64 sourceOffset, Memory<T> dest, int destOffset, int count) where T : unmanaged + { + if (count == 0) + { + return; + } + if (source.Length < (ulong)(sourceOffset + count)) + { + throw new ArgumentException("Source constraints are larger than the source data block", nameof(count)); + } + if(dest.Length < destOffset + count) + { + throw new ArgumentOutOfRangeException(nameof(destOffset), "Destination offset range cannot exceed the size of the destination buffer"); + } + //Get offset to allow large blocks of memory + T* src = source.GetOffset(sourceOffset); + //Pin the memory handle + using MemoryHandle handle = dest.Pin(); + //Byte count + int byteCount = checked(count * sizeof(T)); + //Dest offset + T* dst = ((T*)handle.Pointer) + destOffset; + //Aligned copy + Unsafe.CopyBlock(dst, src, (uint)byteCount); + } + #endregion + + #region Streams + /// <summary> + /// Copies data from one stream to another in specified blocks + /// </summary> + /// <param name="source">Source memory</param> + /// <param name="srcOffset">Source offset</param> + /// <param name="dest">Destination memory</param> + /// <param name="destOffst">Destination offset</param> + /// <param name="count">Number of elements to copy</param> + public static void Copy(Stream source, Int64 srcOffset, Stream dest, Int64 destOffst, Int64 count) + { + if (count == 0) + { + return; + } + if (count < 0) + { + throw new ArgumentException("Count must be a positive integer", nameof(count)); + } + //Seek streams + _ = source.Seek(srcOffset, SeekOrigin.Begin); + _ = dest.Seek(destOffst, SeekOrigin.Begin); + //Create new buffer + using IMemoryHandle<byte> buffer = Shared.Alloc<byte>(count); + Span<byte> buf = buffer.Span; + int total = 0; + do + { + //read from source + int read = source.Read(buf); + //guard + if (read == 0) + { + break; + } + //write read slice to dest + dest.Write(buf[..read]); + //update total read + total += read; + } while (total < count); + } + #endregion + + #region alloc + + /// <summary> + /// Allocates a block of unmanaged, or pooled manaaged memory depending on + /// compilation flags and runtime unamanged allocators. + /// </summary> + /// <typeparam name="T">The unamanged type to allocate</typeparam> + /// <param name="elements">The number of elements of the type within the block</param> + /// <param name="zero">Flag to zero elements during allocation before the method returns</param> + /// <returns>A handle to the block of memory</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static UnsafeMemoryHandle<T> UnsafeAlloc<T>(int elements, bool zero = false) where T : unmanaged + { + if (elements < 0) + { + throw new ArgumentException("Number of elements must be a positive integer", nameof(elements)); + } + if(elements > MAX_UNSAFE_POOL_SIZE || IsRpMallocLoaded) + { + // Alloc from heap + IntPtr block = Shared.Alloc((uint)elements, (uint)sizeof(T), zero); + //Init new handle + return new(Shared, block, elements); + } + else + { + return new(ArrayPool<T>.Shared, elements, zero); + } + } + + /// <summary> + /// Allocates a block of unmanaged, or pooled manaaged memory depending on + /// compilation flags and runtime unamanged allocators. + /// </summary> + /// <typeparam name="T">The unamanged type to allocate</typeparam> + /// <param name="elements">The number of elements of the type within the block</param> + /// <param name="zero">Flag to zero elements during allocation before the method returns</param> + /// <returns>A handle to the block of memory</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static IMemoryHandle<T> SafeAlloc<T>(int elements, bool zero = false) where T: unmanaged + { + if (elements < 0) + { + throw new ArgumentException("Number of elements must be a positive integer", nameof(elements)); + } + + //If the element count is larger than max pool size, alloc from shared heap + if (elements > MAX_UNSAFE_POOL_SIZE) + { + //Alloc from shared heap + return Shared.Alloc<T>(elements, zero); + } + else + { + //Get temp buffer from shared buffer pool + return new VnTempBuffer<T>(elements, zero); + } + } + + #endregion + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/MemoryHandle.cs b/Utils/src/Memory/MemoryHandle.cs new file mode 100644 index 0000000..a09edea --- /dev/null +++ b/Utils/src/Memory/MemoryHandle.cs @@ -0,0 +1,237 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: MemoryHandle.cs +* +* MemoryHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using Microsoft.Win32.SafeHandles; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a wrapper for using umanged memory handles from an assigned <see cref="PrivateHeap"/> for <see cref="UnmanagedType"/> types + /// </summary> + /// <remarks> + /// Handles are configured to address blocks larger than 2GB, + /// so some properties may raise exceptions if large blocks are used. + /// </remarks> + public sealed class MemoryHandle<T> : SafeHandleZeroOrMinusOneIsInvalid, IMemoryHandle<T>, IEquatable<MemoryHandle<T>> where T : unmanaged + { + /// <summary> + /// New <typeparamref name="T"/>* pointing to the base of the allocated block + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + public unsafe T* Base + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => GetOffset(0); + } + /// <summary> + /// New <see cref="IntPtr"/> pointing to the base of the allocated block + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + public unsafe IntPtr BasePtr + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => (IntPtr)GetOffset(0); + } + /// <summary> + /// Gets a span over the entire allocated block + /// </summary> + /// <returns>A <see cref="Span{T}"/> over the internal data</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="OverflowException"></exception> + public unsafe Span<T> Span + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get + { + this.ThrowIfClosed(); + return _length == 0 ? Span<T>.Empty : new Span<T>(Base, IntLength); + } + } + + private readonly bool ZeroMemory; + private readonly IUnmangedHeap Heap; + private ulong _length; + + /// <summary> + /// Number of elements allocated to the current instance + /// </summary> + public ulong Length + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => _length; + } + /// <summary> + /// Number of elements in the memory block casted to an integer + /// </summary> + /// <exception cref="OverflowException"></exception> + public int IntLength + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => checked((int)_length); + } + + /// <summary> + /// Number of bytes allocated to the current instance + /// </summary> + /// <exception cref="OverflowException"></exception> + public unsafe ulong ByteLength + { + //Check for overflows when converting to bytes (should run out of memory before this is an issue, but just incase) + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => checked(_length * (UInt64)sizeof(T)); + } + + /// <summary> + /// Creates a new memory handle, for which is holds ownership, and allocates the number of elements specified on the heap. + /// </summary> + /// <param name="heap">The heap to allocate/deallocate memory from</param> + /// <param name="elements">Number of elements to allocate</param> + /// <param name="zero">Zero all memory during allocations from heap</param> + /// <param name="initial">The initial block of allocated memory to wrap</param> + internal MemoryHandle(IUnmangedHeap heap, IntPtr initial, ulong elements, bool zero) : base(true) + { + //Set element size (always allocate at least 1 object) + _length = elements; + ZeroMemory = zero; + //assign heap ref + Heap = heap; + handle = initial; + } + + /// <summary> + /// Resizes the current handle on the heap + /// </summary> + /// <param name="elements">Positive number of elemnts the current handle should referrence</param> + /// <exception cref="OverflowException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + public unsafe void Resize(ulong elements) + { + this.ThrowIfClosed(); + //Update size (should never be less than inital size) + _length = elements; + //Re-alloc (Zero if required) + try + { + Heap.Resize(ref handle, Length, (ulong)sizeof(T), ZeroMemory); + } + //Catch the disposed exception so we can invalidate the current ptr + catch (ObjectDisposedException) + { + base.handle = IntPtr.Zero; + //Set as invalid so release does not get called + base.SetHandleAsInvalid(); + //Propagate the exception + throw; + } + } + /// <summary> + /// Gets an offset pointer from the base postion to the number of bytes specified. Performs bounds checks + /// </summary> + /// <param name="elements">Number of elements of type to offset</param> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + /// <returns><typeparamref name="T"/> pointer to the memory offset specified</returns> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public unsafe T* GetOffset(ulong elements) + { + if (elements >= _length) + { + throw new ArgumentOutOfRangeException(nameof(elements), "Element offset cannot be larger than allocated size"); + } + this.ThrowIfClosed(); + //Get ptr and offset it + T* bs = ((T*)handle) + elements; + return bs; + } + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + ///<exception cref="ArgumentOutOfRangeException"></exception> + public unsafe MemoryHandle Pin(int elementIndex) + { + //Get ptr and guard checks before adding the referrence + T* ptr = GetOffset((ulong)elementIndex); + + bool addRef = false; + //use the pinned field as success val + DangerousAddRef(ref addRef); + //Create a new system.buffers memory handle from the offset ptr address + return !addRef + ? throw new ObjectDisposedException("Failed to increase referrence count on the memory handle because it was released") + : new MemoryHandle(ptr, pinnable: this); + } + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public void Unpin() + { + //Dec count on release + DangerousRelease(); + } + + + ///<inheritdoc/> + protected override bool ReleaseHandle() + { + //Return result of free + return Heap.Free(ref handle); + } + + + + /// <summary> + /// Determines if the memory blocks are equal by comparing their base addresses. + /// </summary> + /// <param name="other"><see cref="MemoryHandle{T}"/> to compare</param> + /// <returns>true if the block of memory is the same, false if the handle's size does not + /// match or the base addresses do not match even if they point to an overlapping address space</returns> + /// <exception cref="ObjectDisposedException"></exception> + public bool Equals(MemoryHandle<T> other) + { + this.ThrowIfClosed(); + other.ThrowIfClosed(); + return _length == other._length && handle == other.handle; + } + ///<inheritdoc/> + public override bool Equals(object obj) => obj is MemoryHandle<T> oHandle && Equals(oHandle); + ///<inheritdoc/> + public override int GetHashCode() => base.GetHashCode(); + + + ///<inheritdoc/> + public static implicit operator Span<T>(MemoryHandle<T> handle) + { + //If the handle is invalid or closed return an empty span + return handle.IsClosed || handle.IsInvalid || handle._length == 0 ? Span<T>.Empty : handle.Span; + } + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/PrivateBuffersMemoryPool.cs b/Utils/src/Memory/PrivateBuffersMemoryPool.cs new file mode 100644 index 0000000..1e85207 --- /dev/null +++ b/Utils/src/Memory/PrivateBuffersMemoryPool.cs @@ -0,0 +1,67 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: PrivateBuffersMemoryPool.cs +* +* PrivateBuffersMemoryPool.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a <see cref="MemoryPool{T}"/> wrapper for using unmanged <see cref="PrivateHeap"/>s + /// </summary> + /// <typeparam name="T">Unamanged memory type to provide data memory instances from</typeparam> + public sealed class PrivateBuffersMemoryPool<T> : MemoryPool<T> where T : unmanaged + { + private readonly IUnmangedHeap Heap; + + internal PrivateBuffersMemoryPool(IUnmangedHeap heap):base() + { + this.Heap = heap; + } + ///<inheritdoc/> + public override int MaxBufferSize => int.MaxValue; + ///<inheritdoc/> + ///<exception cref="OutOfMemoryException"></exception> + ///<exception cref="ObjectDisposedException"></exception> + ///<exception cref="ArgumentOutOfRangeException"></exception> + public override IMemoryOwner<T> Rent(int minBufferSize = 0) => new SysBufferMemoryManager<T>(Heap, (uint)minBufferSize, false); + + /// <summary> + /// Allocates a new <see cref="MemoryManager{T}"/> of a different data type from the pool + /// </summary> + /// <typeparam name="TDifType">The unmanaged data type to allocate for</typeparam> + /// <param name="minBufferSize">Minumum size of the buffer</param> + /// <returns>The memory owner of a different data type</returns> + public IMemoryOwner<TDifType> Rent<TDifType>(int minBufferSize = 0) where TDifType : unmanaged + { + return new SysBufferMemoryManager<TDifType>(Heap, (uint)minBufferSize, false); + } + ///<inheritdoc/> + protected override void Dispose(bool disposing) + { + //Dispose the heap + Heap.Dispose(); + } + } +} diff --git a/Utils/src/Memory/PrivateHeap.cs b/Utils/src/Memory/PrivateHeap.cs new file mode 100644 index 0000000..5d97506 --- /dev/null +++ b/Utils/src/Memory/PrivateHeap.cs @@ -0,0 +1,184 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: PrivateHeap.cs +* +* PrivateHeap.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Diagnostics; +using System.Runtime.Versioning; +using System.Runtime.InteropServices; + +using DWORD = System.Int64; +using SIZE_T = System.UInt64; +using LPVOID = System.IntPtr; + +namespace VNLib.Utils.Memory +{ + ///<summary> + ///<para> + /// Provides a win32 private heap managed wrapper class + ///</para> + ///</summary> + ///<remarks> + /// <see cref="PrivateHeap"/> implements <see cref="SafeHandle"/> and tracks allocated blocks by its + /// referrence counter. Allocations increment the count, and free's decrement the count, so the heap may + /// be disposed safely + /// </remarks> + [ComVisible(false)] + [SupportedOSPlatform("Windows")] + public sealed class PrivateHeap : UnmanagedHeapBase + { + private const string KERNEL_DLL = "Kernel32"; + + #region Extern + //Heap flags + public const DWORD HEAP_NO_FLAGS = 0x00; + public const DWORD HEAP_GENERATE_EXCEPTIONS = 0x04; + public const DWORD HEAP_NO_SERIALIZE = 0x01; + public const DWORD HEAP_REALLOC_IN_PLACE_ONLY = 0x10; + public const DWORD HEAP_ZERO_MEMORY = 0x08; + + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + private static extern LPVOID HeapAlloc(IntPtr hHeap, DWORD flags, SIZE_T dwBytes); + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + private static extern LPVOID HeapReAlloc(IntPtr hHeap, DWORD dwFlags, LPVOID lpMem, SIZE_T dwBytes); + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool HeapFree(IntPtr hHeap, DWORD dwFlags, LPVOID lpMem); + + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + private static extern LPVOID HeapCreate(DWORD flOptions, SIZE_T dwInitialSize, SIZE_T dwMaximumSize); + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool HeapDestroy(IntPtr hHeap); + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [return: MarshalAs(UnmanagedType.Bool)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + private static extern bool HeapValidate(IntPtr hHeap, DWORD dwFlags, LPVOID lpMem); + [DllImport(KERNEL_DLL, SetLastError = true, ExactSpelling = true)] + [return: MarshalAs(UnmanagedType.U8)] + [DefaultDllImportSearchPaths(DllImportSearchPath.System32)] + private static extern SIZE_T HeapSize(IntPtr hHeap, DWORD flags, LPVOID lpMem); + + #endregion + + /// <summary> + /// Create a new <see cref="PrivateHeap"/> with the specified sizes and flags + /// </summary> + /// <param name="initialSize">Intial size of the heap</param> + /// <param name="maxHeapSize">Maximum size allowed for the heap (disabled = 0, default)</param> + /// <param name="flags">Defalt heap flags to set globally for all blocks allocated by the heap (default = 0)</param> + public static PrivateHeap Create(SIZE_T initialSize, SIZE_T maxHeapSize = 0, DWORD flags = HEAP_NO_FLAGS) + { + //Call create, throw exception if the heap falled to allocate + IntPtr heapHandle = HeapCreate(flags, initialSize, maxHeapSize); + if (heapHandle == IntPtr.Zero) + { + throw new NativeMemoryException("Heap could not be created"); + } +#if TRACE + Trace.WriteLine($"Win32 private heap {heapHandle:x} created"); +#endif + //Heap has been created so we can wrap it + return new(heapHandle); + } + /// <summary> + /// LIFETIME WARNING. Consumes a valid win32 handle and will manage it's lifetime once constructed. + /// Locking and memory blocks will attempt to be allocated from this heap handle. + /// </summary> + /// <param name="win32HeapHandle">An open and valid handle to a win32 private heap</param> + /// <returns>A wrapper around the specified heap</returns> + public static PrivateHeap ConsumeExisting(IntPtr win32HeapHandle) => new (win32HeapHandle); + + private PrivateHeap(IntPtr heapPtr) : base(false, true) => handle = heapPtr; + + /// <summary> + /// Retrieves the size of a memory block allocated from the current heap. + /// </summary> + /// <param name="block">The pointer to a block of memory to get the size of</param> + /// <returns>The size of the block of memory, (SIZE_T)-1 if the operation fails</returns> + public SIZE_T HeapSize(ref LPVOID block) => HeapSize(handle, HEAP_NO_FLAGS, block); + + /// <summary> + /// Validates the specified block of memory within the current heap instance. This function will block hte + /// </summary> + /// <param name="block">Pointer to the block of memory to validate</param> + /// <returns>True if the block is valid, false otherwise</returns> + public bool Validate(ref LPVOID block) + { + bool result; + //Lock the heap before validating + HeapLock.Wait(); + //validate the block on the current heap + result = HeapValidate(handle, HEAP_NO_FLAGS, block); + //Unlock the heap + HeapLock.Release(); + return result; + + } + /// <summary> + /// Validates the current heap instance. The function scans all the memory blocks in the heap and verifies that the heap control structures maintained by + /// the heap manager are in a consistent state. + /// </summary> + /// <returns>If the specified heap or memory block is valid, the return value is nonzero.</returns> + /// <remarks>This can be a consuming operation which will block all allocations</remarks> + public bool Validate() + { + bool result; + //Lock the heap before validating + HeapLock.Wait(); + //validate the entire heap + result = HeapValidate(handle, HEAP_NO_FLAGS, IntPtr.Zero); + //Unlock the heap + HeapLock.Release(); + return result; + } + + ///<inheritdoc/> + protected override bool ReleaseHandle() + { +#if TRACE + Trace.WriteLine($"Win32 private heap {handle:x} destroyed"); +#endif + return HeapDestroy(handle) && base.ReleaseHandle(); + } + ///<inheritdoc/> + protected override sealed LPVOID AllocBlock(ulong elements, ulong size, bool zero) + { + ulong bytes = checked(elements * size); + return HeapAlloc(handle, zero ? HEAP_ZERO_MEMORY : HEAP_NO_FLAGS, bytes); + } + ///<inheritdoc/> + protected override sealed bool FreeBlock(LPVOID block) => HeapFree(handle, HEAP_NO_FLAGS, block); + ///<inheritdoc/> + protected override sealed LPVOID ReAllocBlock(LPVOID block, ulong elements, ulong size, bool zero) + { + ulong bytes = checked(elements * size); + return HeapReAlloc(handle, zero ? HEAP_ZERO_MEMORY : HEAP_NO_FLAGS, block, bytes); + } + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/PrivateString.cs b/Utils/src/Memory/PrivateString.cs new file mode 100644 index 0000000..cd3b1f6 --- /dev/null +++ b/Utils/src/Memory/PrivateString.cs @@ -0,0 +1,185 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: PrivateString.cs +* +* PrivateString.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Diagnostics.CodeAnalysis; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a wrapper class that will have unsafe access to the memory of + /// the specified <see cref="string"/> provided during object creation. + /// </summary> + /// <remarks>The value of the memory the protected string points to is undefined when the instance is disposed</remarks> + public class PrivateString : PrivateStringManager, IEquatable<PrivateString>, IEquatable<string>, ICloneable + { + protected string StrRef => base[0]!; + private readonly bool OwnsReferrence; + + /// <summary> + /// Creates a new <see cref="PrivateString"/> over the specified string and the memory it points to. + /// </summary> + /// <param name="data">The <see cref="string"/> instance pointing to the memory to protect</param> + /// <param name="ownsReferrence">Does the current instance "own" the memory the data parameter points to</param> + /// <remarks>You should no longer reference the input string directly</remarks> + public PrivateString(string data, bool ownsReferrence = true) : base(1) + { + //Create a private string manager to store referrence to string + base[0] = data ?? throw new ArgumentNullException(nameof(data)); + OwnsReferrence = ownsReferrence; + } + + //Create private string from a string + public static explicit operator PrivateString?(string? data) + { + //Allow passing null strings during implicit casting + return data == null ? null : new(data); + } + + public static PrivateString? ToPrivateString(string? value) + { + return value == null ? null : new PrivateString(value, true); + } + + //Cast to string + public static explicit operator string (PrivateString str) + { + //Check if disposed, or return the string + str.Check(); + return str.StrRef; + } + + public static implicit operator ReadOnlySpan<char>(PrivateString str) + { + return str.Disposed ? Span<char>.Empty : str.StrRef.AsSpan(); + } + + /// <summary> + /// Gets the value of the internal string as a <see cref="ReadOnlySpan{T}"/> + /// </summary> + /// <returns>The <see cref="ReadOnlySpan{T}"/> referrence to the internal string</returns> + /// <exception cref="ObjectDisposedException"></exception> + public ReadOnlySpan<char> ToReadOnlySpan() + { + Check(); + return StrRef.AsSpan(); + } + + ///<inheritdoc/> + public bool Equals(string? other) + { + Check(); + return StrRef.Equals(other); + } + ///<inheritdoc/> + public bool Equals(PrivateString? other) + { + Check(); + return other != null && StrRef.Equals(other.StrRef); + } + ///<inheritdoc/> + public override bool Equals(object? other) + { + Check(); + return other is PrivateString otherRef && StrRef.Equals(otherRef); + } + ///<inheritdoc/> + public bool Equals(ReadOnlySpan<char> other) + { + Check(); + return StrRef.AsSpan().SequenceEqual(other); + } + /// <summary> + /// Creates a deep copy of the internal string and returns that copy + /// </summary> + /// <returns>A deep copy of the internal string</returns> + public override string ToString() + { + Check(); + return new(StrRef.AsSpan()); + } + /// <summary> + /// String length + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + public int Length + { + get + { + Check(); + return StrRef.Length; + } + } + /// <summary> + /// Indicates whether the underlying string is null or an empty string ("") + /// </summary> + /// <param name="ps"></param> + /// <returns>True if the parameter is null, or an empty string (""). False otherwise</returns> + public static bool IsNullOrEmpty([NotNullWhen(false)] PrivateString? ps) => ps == null|| ps.Length == 0; + + /// <summary> + /// The hashcode of the underlying string + /// </summary> + /// <returns></returns> + public override int GetHashCode() + { + Check(); + return StrRef.GetHashCode(); + } + + /// <summary> + /// Creates a new deep copy of the current instance that is an independent <see cref="PrivateString"/> + /// </summary> + /// <returns>The new <see cref="PrivateString"/> instance</returns> + /// <exception cref="ObjectDisposedException"></exception> + public override object Clone() + { + Check(); + //Copy all contents of string to another reference + string clone = new (StrRef.AsSpan()); + //return a new private string + return new PrivateString(clone, true); + } + + ///<inheritdoc/> + protected override void Free() + { + Erase(); + } + + /// <summary> + /// Erases the contents of the internal CLR string + /// </summary> + public void Erase() + { + //Only dispose the instance if we own the memory + if (OwnsReferrence && !Disposed) + { + base.Free(); + } + } + + + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/PrivateStringManager.cs b/Utils/src/Memory/PrivateStringManager.cs new file mode 100644 index 0000000..9ed8f5f --- /dev/null +++ b/Utils/src/Memory/PrivateStringManager.cs @@ -0,0 +1,117 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: PrivateStringManager.cs +* +* PrivateStringManager.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// When inherited by a class, provides a safe string storage that zeros a CLR string memory on disposal + /// </summary> + public class PrivateStringManager : VnDisposeable, ICloneable + { + /// <summary> + /// Strings to be cleared when exiting + /// </summary> + private readonly string?[] ProtectedElements; + /// <summary> + /// Gets or sets a string referrence into the protected elements store + /// </summary> + /// <param name="index"></param> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + /// <exception cref="ObjectDisposedException"></exception> + /// <returns>Referrence to string associated with the index</returns> + protected string? this[int index] + { + get + { + Check(); + return ProtectedElements[index]; + } + set + { + Check(); + //Check to see if the string has been interned + if (!string.IsNullOrEmpty(value) && string.IsInterned(value) != null) + { + throw new ArgumentException($"The specified string has been CLR interned and cannot be stored in {nameof(PrivateStringManager)}"); + } + //Clear the old value before setting the new one + if (!string.IsNullOrEmpty(ProtectedElements[index])) + { + Memory.UnsafeZeroMemory<char>(ProtectedElements[index]); + } + //set new value + ProtectedElements[index] = value; + } + } + /// <summary> + /// Create a new instance with fixed array size + /// </summary> + /// <param name="elements">Number of elements to protect</param> + public PrivateStringManager(int elements) + { + //Allocate the string array + ProtectedElements = new string[elements]; + } + ///<inheritdoc/> + protected override void Free() + { + //Zero all strings specified + for (int i = 0; i < ProtectedElements.Length; i++) + { + if (!string.IsNullOrEmpty(ProtectedElements[i])) + { + //Zero the string memory + Memory.UnsafeZeroMemory<char>(ProtectedElements[i]); + //Set to null + ProtectedElements[i] = null; + } + } + } + + /// <summary> + /// Creates a deep copy for a new independent <see cref="PrivateStringManager"/> + /// </summary> + /// <returns>A new independent <see cref="PrivateStringManager"/> instance</returns> + /// <remarks>Be careful duplicating large instances, and make sure clones are properly disposed if necessary</remarks> + /// <exception cref="ObjectDisposedException"></exception> + public virtual object Clone() + { + Check(); + PrivateStringManager other = new (ProtectedElements.Length); + //Copy all strings to the other instance + for(int i = 0; i < ProtectedElements.Length; i++) + { + //Copy all strings and store their copies in the new array + other.ProtectedElements[i] = this.ProtectedElements[i].AsSpan().ToString(); + } + //return the new copy + return other; + } + } +} diff --git a/Utils/src/Memory/ProcessHeap.cs b/Utils/src/Memory/ProcessHeap.cs new file mode 100644 index 0000000..4f06d52 --- /dev/null +++ b/Utils/src/Memory/ProcessHeap.cs @@ -0,0 +1,82 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ProcessHeap.cs +* +* ProcessHeap.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Diagnostics; +using System.Runtime.InteropServices; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a <see cref="IUnmangedHeap"/> wrapper for the <see cref="Marshal"/> virtualalloc + /// global heap methods + /// </summary> + [ComVisible(false)] + public unsafe class ProcessHeap : VnDisposeable, IUnmangedHeap + { + /// <summary> + /// Initalizes a new global (cross platform) process heap + /// </summary> + public ProcessHeap() + { +#if TRACE + Trace.WriteLine($"Default heap instnace created {GetHashCode():x}"); +#endif + } + + ///<inheritdoc/> + ///<exception cref="OverflowException"></exception> + ///<exception cref="OutOfMemoryException"></exception> + public IntPtr Alloc(ulong elements, ulong size, bool zero) + { + return zero + ? (IntPtr)NativeMemory.AllocZeroed((nuint)elements, (nuint)size) + : (IntPtr)NativeMemory.Alloc((nuint)elements, (nuint)size); + } + ///<inheritdoc/> + public bool Free(ref IntPtr block) + { + //Free native mem from ptr + NativeMemory.Free(block.ToPointer()); + block = IntPtr.Zero; + return true; + } + ///<inheritdoc/> + protected override void Free() + { +#if TRACE + Trace.WriteLine($"Default heap instnace disposed {GetHashCode():x}"); +#endif + } + ///<inheritdoc/> + ///<exception cref="OverflowException"></exception> + ///<exception cref="OutOfMemoryException"></exception> + public void Resize(ref IntPtr block, ulong elements, ulong size, bool zero) + { + nuint bytes = checked((nuint)(elements * size)); + IntPtr old = block; + block = (IntPtr)NativeMemory.Realloc(old.ToPointer(), bytes); + } + } +} diff --git a/Utils/src/Memory/RpMallocPrivateHeap.cs b/Utils/src/Memory/RpMallocPrivateHeap.cs new file mode 100644 index 0000000..70c8a7f --- /dev/null +++ b/Utils/src/Memory/RpMallocPrivateHeap.cs @@ -0,0 +1,279 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: RpMallocPrivateHeap.cs +* +* RpMallocPrivateHeap.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Diagnostics; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using size_t = System.UInt64; +using LPVOID = System.IntPtr; +using LPHEAPHANDLE = System.IntPtr; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// A wrapper class for cross platform RpMalloc implementation. + /// </summary> + [ComVisible(false)] + public sealed class RpMallocPrivateHeap : UnmanagedHeapBase + { + const string DLL_NAME = "rpmalloc"; + + #region statics + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern int rpmalloc_initialize(); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_finalize(); + + //Heap api + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPHEAPHANDLE rpmalloc_heap_acquire(); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_heap_release(LPHEAPHANDLE heap); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc_heap_alloc(LPHEAPHANDLE heap, size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc_heap_aligned_alloc(LPHEAPHANDLE heap, size_t alignment, size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc_heap_calloc(LPHEAPHANDLE heap, size_t num, size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc_heap_aligned_calloc(LPHEAPHANDLE heap, size_t alignment, size_t num, size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc_heap_realloc(LPHEAPHANDLE heap, LPVOID ptr, size_t size, nuint flags); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc_heap_aligned_realloc(LPHEAPHANDLE heap, LPVOID ptr, size_t alignment, size_t size, nuint flags); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_heap_free(LPHEAPHANDLE heap, LPVOID ptr); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_heap_free_all(LPHEAPHANDLE heap); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_heap_thread_set_current(LPHEAPHANDLE heap); + + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_thread_initialize(); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern int rpmalloc_is_thread_initialized(); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpmalloc_thread_finalize(int release_caches); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpmalloc(size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rpcalloc(size_t num, size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern LPVOID rprealloc(LPVOID ptr, size_t size); + [DllImport(DLL_NAME, ExactSpelling = true)] + [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)] + static extern void rpfree(LPVOID ptr); + + #endregion + + private class RpMallocGlobalHeap : IUnmangedHeap + { + IntPtr IUnmangedHeap.Alloc(ulong elements, ulong size, bool zero) + { + return RpMalloc(elements, (nuint)size, zero); + } + + //Global heap does not need to be disposed + void IDisposable.Dispose() + { } + + bool IUnmangedHeap.Free(ref IntPtr block) + { + //Free the block + RpFree(ref block); + return true; + } + + void IUnmangedHeap.Resize(ref IntPtr block, ulong elements, ulong size, bool zero) + { + //Try to resize the block + IntPtr resize = RpRealloc(block, elements, (nuint)size); + + if (resize == IntPtr.Zero) + { + throw new NativeMemoryOutOfMemoryException("Failed to resize the block"); + } + //assign ptr + block = resize; + } + } + + /// <summary> + /// <para> + /// A <see cref="IUnmangedHeap"/> API for the RPMalloc library if loaded. + /// </para> + /// <para> + /// This heap is thread safe and may be converted to a <see cref="MemoryManager{T}"/> + /// infinitley and disposed safely. + /// </para> + /// <para> + /// If the native library is not loaded, calls to this API will throw a <see cref="DllNotFoundException"/>. + /// </para> + /// </summary> + public static IUnmangedHeap GlobalHeap { get; } = new RpMallocGlobalHeap(); + + /// <summary> + /// <para> + /// Initializes RpMalloc for the current thread and alloctes a block of memory + /// </para> + /// </summary> + /// <param name="elements">The number of elements to allocate</param> + /// <param name="size">The number of bytes per element type (aligment)</param> + /// <param name="zero">Zero the block of memory before returning</param> + /// <returns>A pointer to the block, (zero if failed)</returns> + public static LPVOID RpMalloc(size_t elements, nuint size, bool zero) + { + //See if the current thread has been initialized + if (rpmalloc_is_thread_initialized() == 0) + { + //Initialize the current thread + rpmalloc_thread_initialize(); + } + //Alloc block + LPVOID block; + if (zero) + { + block = rpcalloc(elements, size); + } + else + { + //Calculate the block size + ulong blockSize = checked(elements * size); + block = rpmalloc(blockSize); + } + return block; + } + + /// <summary> + /// Frees a block of memory allocated by RpMalloc + /// </summary> + /// <param name="block">A ref to the pointer of the block to free</param> + public static void RpFree(ref LPVOID block) + { + if (block != IntPtr.Zero) + { + rpfree(block); + block = IntPtr.Zero; + } + } + + /// <summary> + /// Attempts to re-allocate the specified block on the global heap + /// </summary> + /// <param name="block">A pointer to a previously allocated block of memory</param> + /// <param name="elements">The number of elements in the block</param> + /// <param name="size">The number of bytes in the element</param> + /// <returns>A pointer to the new block if the reallocation succeeded, null if the resize failed</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="OverflowException"></exception> + public static LPVOID RpRealloc(LPVOID block, size_t elements, nuint size) + { + if(block == IntPtr.Zero) + { + throw new ArgumentException("The supplied block is not valid", nameof(block)); + } + //Calc new block size + size_t blockSize = checked(elements * size); + return rprealloc(block, blockSize); + } + + #region instance + + /// <summary> + /// Initializes a new RpMalloc first class heap to allocate memory blocks from + /// </summary> + /// <param name="zeroAll">A global flag to zero all blocks of memory allocated</param> + /// <exception cref="NativeMemoryOutOfMemoryException"></exception> + public RpMallocPrivateHeap(bool zeroAll):base(zeroAll, true) + { + //Alloc the heap + handle = rpmalloc_heap_acquire(); + if(IsInvalid) + { + throw new NativeMemoryException("Failed to aquire a new heap"); + } +#if TRACE + Trace.WriteLine($"RPMalloc heap {handle:x} created"); +#endif + } + ///<inheritdoc/> + protected override bool ReleaseHandle() + { +#if TRACE + Trace.WriteLine($"RPMalloc heap {handle:x} destroyed"); +#endif + //Release all heap memory + rpmalloc_heap_free_all(handle); + //Destroy the heap + rpmalloc_heap_release(handle); + //Release base + return base.ReleaseHandle(); + } + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected sealed override LPVOID AllocBlock(ulong elements, ulong size, bool zero) + { + //Alloc or calloc and initalize + return zero ? rpmalloc_heap_calloc(handle, elements, size) : rpmalloc_heap_alloc(handle, checked(size * elements)); + } + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected sealed override bool FreeBlock(LPVOID block) + { + //Free block + rpmalloc_heap_free(handle, block); + return true; + } + ///<inheritdoc/> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected sealed override LPVOID ReAllocBlock(LPVOID block, ulong elements, ulong size, bool zero) + { + //Realloc + return rpmalloc_heap_realloc(handle, block, checked(elements * size), 0); + } + #endregion + } +} diff --git a/Utils/src/Memory/SubSequence.cs b/Utils/src/Memory/SubSequence.cs new file mode 100644 index 0000000..3800fb5 --- /dev/null +++ b/Utils/src/Memory/SubSequence.cs @@ -0,0 +1,113 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SubSequence.cs +* +* SubSequence.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Represents a subset (or window) of data within a <see cref="MemoryHandle{T}"/> + /// </summary> + /// <typeparam name="T">The unmanaged type to wrap</typeparam> + public readonly struct SubSequence<T> : IEquatable<SubSequence<T>> where T: unmanaged + { + private readonly MemoryHandle<T> _handle; + /// <summary> + /// The number of elements in the current window + /// </summary> + public readonly int Size { get; } + + /// <summary> + /// Creates a new <see cref="SubSequence{T}"/> to the handle to get a window of the block + /// </summary> + /// <param name="block"></param> + /// <param name="offset"></param> + /// <param name="size"></param> +#if TARGET_64_BIT + public SubSequence(MemoryHandle<T> block, ulong offset, int size) +#else + public SubSequence(MemoryHandle<T> block, int offset, int size) +#endif + { + _offset = offset; + Size = size >= 0 ? size : throw new ArgumentOutOfRangeException(nameof(size)); + _handle = block ?? throw new ArgumentNullException(nameof(block)); + } + + +#if TARGET_64_BIT + private readonly ulong _offset; +#else + private readonly int _offset; +#endif + /// <summary> + /// Gets a <see cref="Span{T}"/> that is offset from the base of the handle + /// </summary> + /// <exception cref="ArgumentOutOfRangeException"></exception> + +#if TARGET_64_BIT + public readonly Span<T> Span => Size > 0 ? _handle.GetOffsetSpan(_offset, Size) : Span<T>.Empty; +#else + public readonly Span<T> Span => Size > 0 ? _handle.Span.Slice(_offset, Size) : Span<T>.Empty; +#endif + + /// <summary> + /// Slices the current sequence into a smaller <see cref="SubSequence{T}"/> + /// </summary> + /// <param name="offset">The relative offset from the current window offset</param> + /// <param name="size">The size of the block</param> + /// <returns>A <see cref="SubSequence{T}"/> of the current sequence</returns> + public readonly SubSequence<T> Slice(uint offset, int size) => new (_handle, _offset + checked((int)offset), size); + + /// <summary> + /// Returns the signed 32-bit hashcode + /// </summary> + /// <returns>A signed 32-bit integer that represents the hashcode for the current instance</returns> + /// <exception cref="ObjectDisposedException"></exception> + public readonly override int GetHashCode() => _handle.GetHashCode() + _offset.GetHashCode(); + + ///<inheritdoc/> + public readonly bool Equals(SubSequence<T> other) => Span.SequenceEqual(other.Span); + + ///<inheritdoc/> + public readonly override bool Equals(object? obj) => obj is SubSequence<T> other && Equals(other); + + /// <summary> + /// Determines if two <see cref="SubSequence{T}"/> are equal + /// </summary> + /// <param name="left"></param> + /// <param name="right"></param> + /// <returns>True if the sequences are equal, false otherwise</returns> + public static bool operator ==(SubSequence<T> left, SubSequence<T> right) => left.Equals(right); + /// <summary> + /// Determines if two <see cref="SubSequence{T}"/> are not equal + /// </summary> + /// <param name="left"></param> + /// <param name="right"></param> + /// <returns>True if the sequences are not equal, false otherwise</returns> + public static bool operator !=(SubSequence<T> left, SubSequence<T> right) => !left.Equals(right); + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/SysBufferMemoryManager.cs b/Utils/src/Memory/SysBufferMemoryManager.cs new file mode 100644 index 0000000..040467f --- /dev/null +++ b/Utils/src/Memory/SysBufferMemoryManager.cs @@ -0,0 +1,102 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SysBufferMemoryManager.cs +* +* SysBufferMemoryManager.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides an unmanaged System.Buffers integration with zero-cost pinning. Uses <see cref="MemoryHandle{T}"/> + /// as a memory provider which implements a <see cref="System.Runtime.InteropServices.SafeHandle"/> + /// </summary> + /// <typeparam name="T">Unmanaged memory type</typeparam> + public sealed class SysBufferMemoryManager<T> : MemoryManager<T> where T :unmanaged + { + private readonly IMemoryHandle<T> BackingMemory; + private readonly bool _ownsHandle; + + /// <summary> + /// Consumes an exisitng <see cref="MemoryHandle{T}"/> to provide <see cref="Memory"/> wrappers. + /// The handle should no longer be referrenced directly + /// </summary> + /// <param name="existingHandle">The existing handle to consume</param> + /// <param name="ownsHandle">A value that indicates if the memory manager owns the handle reference</param> + internal SysBufferMemoryManager(IMemoryHandle<T> existingHandle, bool ownsHandle) + { + BackingMemory = existingHandle; + _ownsHandle = ownsHandle; + } + + /// <summary> + /// Allocates a fized size buffer from the specified unmanaged <see cref="PrivateHeap"/> + /// </summary> + /// <param name="heap">The heap to perform allocations from</param> + /// <param name="elements">The number of elements to allocate</param> + /// <param name="zero">Zero allocations</param> + public SysBufferMemoryManager(IUnmangedHeap heap, ulong elements, bool zero) + { + BackingMemory = heap.Alloc<T>(elements, zero); + _ownsHandle = true; + } + + ///<inheritdoc/> + protected override bool TryGetArray(out ArraySegment<T> segment) + { + //Always false since no array is available + segment = default; + return false; + } + + ///<inheritdoc/> + ///<exception cref="OverflowException"></exception> + ///<exception cref="ObjectDisposedException"></exception> + public override Span<T> GetSpan() => BackingMemory.Span; + + /// <summary> + /// <inheritdoc/> + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public unsafe override MemoryHandle Pin(int elementIndex = 0) + { + return BackingMemory.Pin(elementIndex); + } + + ///<inheritdoc/> + public override void Unpin() + {} + + ///<inheritdoc/> + protected override void Dispose(bool disposing) + { + if (_ownsHandle) + { + BackingMemory.Dispose(); + } + } + } +} diff --git a/Utils/src/Memory/UnmanagedHeapBase.cs b/Utils/src/Memory/UnmanagedHeapBase.cs new file mode 100644 index 0000000..5c92aff --- /dev/null +++ b/Utils/src/Memory/UnmanagedHeapBase.cs @@ -0,0 +1,185 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: UnmanagedHeapBase.cs +* +* UnmanagedHeapBase.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Threading; +using System.Runtime.InteropServices; + +using Microsoft.Win32.SafeHandles; + +using size_t = System.UInt64; +using LPVOID = System.IntPtr; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a synchronized base methods for accessing unmanaged memory. Implements <see cref="SafeHandle"/> + /// for safe disposal of heaps + /// </summary> + public abstract class UnmanagedHeapBase : SafeHandleZeroOrMinusOneIsInvalid, IUnmangedHeap + { + /// <summary> + /// The heap synchronization handle + /// </summary> + protected readonly SemaphoreSlim HeapLock; + /// <summary> + /// The global heap zero flag + /// </summary> + protected readonly bool GlobalZero; + + /// <summary> + /// Initalizes the unmanaged heap base class (init synchronization handle) + /// </summary> + /// <param name="globalZero">A global flag to zero all blocks of memory during allocation</param> + /// <param name="ownsHandle">A flag that indicates if the handle is owned by the instance</param> + protected UnmanagedHeapBase(bool globalZero, bool ownsHandle) : base(ownsHandle) + { + HeapLock = new(1, 1); + GlobalZero = globalZero; + } + + ///<inheritdoc/> + ///<remarks>Increments the handle count</remarks> + ///<exception cref="OutOfMemoryException"></exception> + ///<exception cref="ObjectDisposedException"></exception> + public LPVOID Alloc(size_t elements, size_t size, bool zero) + { + //Force zero if global flag is set + zero |= GlobalZero; + bool handleCountIncremented = false; + //Increment handle count to prevent premature release + DangerousAddRef(ref handleCountIncremented); + //Failed to increment ref count, class has been disposed + if (!handleCountIncremented) + { + throw new ObjectDisposedException("The handle has been released"); + } + try + { + //wait for lock + HeapLock.Wait(); + //Alloc block + LPVOID block = AllocBlock(elements, size, zero); + //release lock + HeapLock.Release(); + //Check if block was allocated + return block != IntPtr.Zero ? block : throw new NativeMemoryOutOfMemoryException("Failed to allocate the requested block"); + } + catch + { + //Decrement handle count since allocation failed + DangerousRelease(); + throw; + } + } + ///<inheritdoc/> + ///<remarks>Decrements the handle count</remarks> + public bool Free(ref LPVOID block) + { + bool result; + //If disposed, set the block handle to zero and exit to avoid raising exceptions during finalization + if (IsClosed || IsInvalid) + { + block = IntPtr.Zero; + return true; + } + //wait for lock + HeapLock.Wait(); + //Free block + result = FreeBlock(block); + //Release lock before releasing handle + HeapLock.Release(); + //Decrement handle count + DangerousRelease(); + //set block to invalid + block = IntPtr.Zero; + return result; + } + ///<inheritdoc/> + ///<exception cref="OutOfMemoryException"></exception> + ///<exception cref="ObjectDisposedException"></exception> + public void Resize(ref LPVOID block, size_t elements, size_t size, bool zero) + { + //wait for lock + HeapLock.Wait(); + /* + * Realloc may return a null pointer if allocation fails + * so check the results and only assign the block pointer + * if the result is valid. Otherwise pointer block should + * be left untouched + */ + LPVOID newBlock = ReAllocBlock(block, elements, size, zero); + //release lock + HeapLock.Release(); + //Check block + if (newBlock == IntPtr.Zero) + { + throw new NativeMemoryOutOfMemoryException("The memory block could not be resized"); + } + //Set the new block + block = newBlock; + } + + ///<inheritdoc/> + protected override bool ReleaseHandle() + { + HeapLock.Dispose(); + return true; + } + + /// <summary> + /// Allocates a block of memory from the heap + /// </summary> + /// <param name="elements">The number of elements within the block</param> + /// <param name="size">The size of the element type (in bytes)</param> + /// <param name="zero">A flag to zero the allocated block</param> + /// <returns>A pointer to the allocated block</returns> + protected abstract LPVOID AllocBlock(size_t elements, size_t size, bool zero); + /// <summary> + /// Frees a previously allocated block of memory + /// </summary> + /// <param name="block">The block to free</param> + protected abstract bool FreeBlock(LPVOID block); + /// <summary> + /// Resizes the previously allocated block of memory on the current heap + /// </summary> + /// <param name="block">The prevously allocated block</param> + /// <param name="elements">The new number of elements within the block</param> + /// <param name="size">The size of the element type (in bytes)</param> + /// <param name="zero">A flag to indicate if the new region of the block should be zeroed</param> + /// <returns>A pointer to the same block, but resized, null if the allocation falied</returns> + /// <remarks> + /// Heap base relies on the block pointer to remain unchanged if the resize fails so the + /// block is still valid, and the return value is used to determine if the resize was successful + /// </remarks> + protected abstract LPVOID ReAllocBlock(LPVOID block, size_t elements, size_t size, bool zero); + ///<inheritdoc/> + public override int GetHashCode() => handle.GetHashCode(); + ///<inheritdoc/> + public override bool Equals(object? obj) + { + return obj is UnmanagedHeapBase heap && !heap.IsInvalid && !heap.IsClosed && handle == heap.handle; + } + } +} diff --git a/Utils/src/Memory/UnsafeMemoryHandle.cs b/Utils/src/Memory/UnsafeMemoryHandle.cs new file mode 100644 index 0000000..b05ad40 --- /dev/null +++ b/Utils/src/Memory/UnsafeMemoryHandle.cs @@ -0,0 +1,231 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: UnsafeMemoryHandle.cs +* +* UnsafeMemoryHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; +using System.Diagnostics.CodeAnalysis; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Represents an unsafe handle to managed/unmanaged memory that should be used cautiously. + /// A referrence counter is not maintained. + /// </summary> + /// <typeparam name="T">Unmanaged memory type</typeparam> + [StructLayout(LayoutKind.Sequential)] + public readonly struct UnsafeMemoryHandle<T> : IMemoryHandle<T>, IEquatable<UnsafeMemoryHandle<T>> where T : unmanaged + { + private enum HandleType + { + None, + Pool, + PrivateHeap + } + + private readonly T[]? _poolArr; + private readonly IntPtr _memoryPtr; + private readonly ArrayPool<T>? _pool; + private readonly IUnmangedHeap? _heap; + private readonly HandleType _handleType; + private readonly int _length; + + ///<inheritdoc/> + public readonly unsafe Span<T> Span + { + [MethodImpl(MethodImplOptions.AggressiveInlining)] + get => _handleType == HandleType.Pool ? _poolArr.AsSpan(0, IntLength) : new (_memoryPtr.ToPointer(), IntLength); + } + ///<inheritdoc/> + public readonly int IntLength => _length; + ///<inheritdoc/> + public readonly ulong Length => (ulong)_length; + + /// <summary> + /// Creates an empty <see cref="UnsafeMemoryHandle{T}"/> + /// </summary> + public UnsafeMemoryHandle() + { + _pool = null; + _heap = null; + _poolArr = null; + _memoryPtr = IntPtr.Zero; + _handleType = HandleType.None; + _length = 0; + } + + /// <summary> + /// Inializes a new <see cref="UnsafeMemoryHandle{T}"/> using the specified + /// <see cref="ArrayPool{T}"/> + /// </summary> + /// <param name="elements">The number of elements to store</param> + /// <param name="zero">Zero initial contents?</param> + /// <param name="pool">The explicit pool to alloc buffers from</param> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public unsafe UnsafeMemoryHandle(ArrayPool<T> pool, int elements, bool zero) + { + if (elements < 0) + { + throw new ArgumentOutOfRangeException(nameof(elements)); + } + //Pool is required + _pool = pool ?? throw new ArgumentNullException(nameof(pool)); + //Rent the array from the pool and hold referrence to it + _poolArr = pool.Rent(elements, zero); + //Cant store ref to array becase GC can move it + _memoryPtr = IntPtr.Zero; + //Set pool handle type + _handleType = HandleType.Pool; + //No heap being loaded + _heap = null; + _length = elements; + } + + /// <summary> + /// Intializes a new <see cref="UnsafeMemoryHandle{T}"/> for block of memory allocated from + /// an <see cref="IUnmangedHeap"/> + /// </summary> + /// <param name="heap">The heap the initial memory block belongs to</param> + /// <param name="initial">A pointer to the unmanaged memory block</param> + /// <param name="elements">The number of elements this block points to</param> + internal UnsafeMemoryHandle(IUnmangedHeap heap, IntPtr initial, int elements) + { + _pool = null; + _poolArr = null; + _heap = heap; + _length = elements; + _memoryPtr = initial; + _handleType = HandleType.PrivateHeap; + } + + /// <summary> + /// Releases memory back to the pool or heap from which is was allocated. + /// </summary> + /// <remarks>After this method is called, this handle points to invalid memory</remarks> + public readonly void Dispose() + { + switch (_handleType) + { + case HandleType.Pool: + { + //Return array to pool + _pool!.Return(_poolArr!); + } + break; + case HandleType.PrivateHeap: + { + IntPtr unalloc = _memoryPtr; + //Free the unmanaged handle + _heap!.Free(ref unalloc); + } + break; + } + } + + ///<inheritdoc/> + public readonly override int GetHashCode() => _handleType == HandleType.Pool ? _poolArr!.GetHashCode() : _memoryPtr.GetHashCode(); + ///<inheritdoc/> + public readonly unsafe MemoryHandle Pin(int elementIndex) + { + //Guard + if (elementIndex < 0 || elementIndex >= IntLength) + { + throw new ArgumentOutOfRangeException(nameof(elementIndex)); + } + + if (_handleType == HandleType.Pool) + { + //Pin the array + GCHandle arrHandle = GCHandle.Alloc(_poolArr, GCHandleType.Pinned); + //Get array base address + void* basePtr = (void*)arrHandle.AddrOfPinnedObject(); + //Get element offset + void* indexOffet = Unsafe.Add<T>(basePtr, elementIndex); + return new (indexOffet, arrHandle); + } + else + { + //Get offset pointer and pass self as pinnable argument, (nothing happens but support it) + void* basePtr = Unsafe.Add<T>(_memoryPtr.ToPointer(), elementIndex); + //Unmanaged memory is always pinned, so no need to pass this as IPinnable, since it will cause a box + return new (basePtr); + } + } + ///<inheritdoc/> + public readonly void Unpin() + { + //Nothing to do since gc handle takes care of array, and unmanaged pointers are not pinned + } + + /// <summary> + /// Determines if the other handle represents the same memory block as the + /// current handle. + /// </summary> + /// <param name="other">The other handle to test</param> + /// <returns>True if the other handle points to the same block of memory as the current handle</returns> + public readonly bool Equals(UnsafeMemoryHandle<T> other) + { + return _handleType == other._handleType && Length == other.Length && GetHashCode() == other.GetHashCode(); + } + + /// <summary> + /// Override for object equality operator, will cause boxing + /// for structures + /// </summary> + /// <param name="obj">The other object to compare</param> + /// <returns> + /// True if the passed object is of type <see cref="UnsafeMemoryHandle{T}"/> + /// and uses the structure equality operator + /// false otherwise. + /// </returns> + public readonly override bool Equals([NotNullWhen(true)] object? obj) => obj is UnsafeMemoryHandle<T> other && Equals(other); + + /// <summary> + /// Casts the handle to it's <see cref="Span{T}"/> representation + /// </summary> + /// <param name="handle">the handle to cast</param> + public static implicit operator Span<T>(in UnsafeMemoryHandle<T> handle) => handle.Span; + + /// <summary> + /// Equality overload + /// </summary> + /// <param name="left"></param> + /// <param name="right"></param> + /// <returns>True if handles are equal, flase otherwise</returns> + public static bool operator ==(in UnsafeMemoryHandle<T> left, in UnsafeMemoryHandle<T> right) => left.Equals(right); + /// <summary> + /// Equality overload + /// </summary> + /// <param name="left"></param> + /// <param name="right"></param> + /// <returns>True if handles are equal, flase otherwise</returns> + public static bool operator !=(in UnsafeMemoryHandle<T> left, in UnsafeMemoryHandle<T> right) => !left.Equals(right); + + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/VnString.cs b/Utils/src/Memory/VnString.cs new file mode 100644 index 0000000..7fa0c5a --- /dev/null +++ b/Utils/src/Memory/VnString.cs @@ -0,0 +1,497 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnString.cs +* +* VnString.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Buffers; +using System.ComponentModel; +using System.Threading.Tasks; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using VNLib.Utils.IO; +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + + /// <summary> + /// Provides an immutable character buffer stored on an unmanged heap. Contains handles to unmanged memory, and should be disposed + /// </summary> + [ComVisible(false)] + [ImmutableObject(true)] + public sealed class VnString : VnDisposeable, IEquatable<VnString>, IEquatable<string>, IEquatable<char[]>, IComparable<VnString>, IComparable<string> + { + private readonly MemoryHandle<char>? Handle; + + private readonly SubSequence<char> _stringSequence; + + /// <summary> + /// The number of unicode characters the current instance can reference + /// </summary> + public int Length => _stringSequence.Size; + /// <summary> + /// Gets a value indicating if the current instance is empty + /// </summary> + public bool IsEmpty => Length == 0; + + private VnString(SubSequence<char> sequence) + { + _stringSequence = sequence; + } + + private VnString( + MemoryHandle<char> handle, +#if TARGET_64_BIT + ulong start, +#else + int start, +#endif + int length) + { + Handle = handle ?? throw new ArgumentNullException(nameof(handle)); + //get sequence + _stringSequence = handle.GetSubSequence(start, length); + } + + /// <summary> + /// Creates and empty <see cref="VnString"/>, not particularly usefull, just and empty instance. + /// </summary> + public VnString() + { + //Default string sequence is empty and does not hold any memory + } + /// <summary> + /// Creates a new <see cref="VnString"/> around a <see cref="ReadOnlySpan{T}"/> or a <see cref="string"/> of data + /// </summary> + /// <param name="data"><see cref="ReadOnlySpan{T}"/> of data to replicate</param> + /// <exception cref="OutOfMemoryException"></exception> + public VnString(ReadOnlySpan<char> data) + { + //Create new handle with enough size (heap) + Handle = Memory.Shared.Alloc<char>(data.Length); + //Copy + Memory.Copy(data, Handle, 0); + //Get subsequence over the whole copy of data + _stringSequence = Handle.GetSubSequence(0, data.Length); + } + /// <summary> + /// Allocates a temporary buffer to read data from the stream until the end of the stream is reached. + /// Decodes data from the user-specified encoding + /// </summary> + /// <param name="stream">Active stream of data to decode to a string</param> + /// <param name="encoding"><see cref="Encoding"/> to use for decoding</param> + /// <param name="bufferSize">The size of the buffer to allocate during copying</param> + /// <returns>The new <see cref="VnString"/> instance</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="OverflowException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + /// <exception cref="InvalidOperationException"></exception> + public static VnString FromStream(Stream stream, Encoding encoding, uint bufferSize) + { + //Make sure the stream is readable + if (!stream.CanRead) + { + throw new InvalidOperationException(); + } + //See if the stream is a vn memory stream + if (stream is VnMemoryStream vnms) + { + //Get the number of characters + int numChars = encoding.GetCharCount(vnms.AsSpan()); + //New handle + MemoryHandle<char> charBuffer = Memory.Shared.Alloc<char>(numChars); + try + { + //Write characters to character buffer + _ = encoding.GetChars(vnms.AsSpan(), charBuffer); + //Consume the new handle + return ConsumeHandle(charBuffer, 0, numChars); + } + catch + { + //If an error occured, dispose the buffer + charBuffer.Dispose(); + throw; + } + } + //Need to read from the stream old school with buffers + else + { + //Create a new char bufer that will expand dyanmically + MemoryHandle<char> charBuffer = Memory.Shared.Alloc<char>(bufferSize); + //Allocate a binary buffer + MemoryHandle<byte> binBuffer = Memory.Shared.Alloc<byte>(bufferSize); + try + { + int length = 0; + //span ref to bin buffer + Span<byte> buffer = binBuffer; + //Run in checked context for overflows + checked + { + do + { + //read + int read = stream.Read(buffer); + //guard + if (read <= 0) + { + break; + } + //Slice into only the read data + ReadOnlySpan<byte> readbytes = buffer[..read]; + //get num chars + int numChars = encoding.GetCharCount(readbytes); + //Guard for overflow + if (((ulong)(numChars + length)) >= int.MaxValue) + { + throw new OverflowException(); + } + //Re-alloc buffer + charBuffer.ResizeIfSmaller(length + numChars); + //Decode and update position + _= encoding.GetChars(readbytes, charBuffer.Span.Slice(length, numChars)); + //Update char count + length += numChars; + } while (true); + } + return ConsumeHandle(charBuffer, 0, length); + } + catch + { + //Free the memory allocated + charBuffer.Dispose(); + //We still want the exception to be propagated! + throw; + } + finally + { + //Dispose the binary buffer + binBuffer.Dispose(); + } + } + } + /// <summary> + /// Creates a new Vnstring from the <see cref="MemoryHandle{T}"/> buffer provided. This function "consumes" + /// a handle, meaning it now takes ownsership of the the memory it points to. + /// </summary> + /// <param name="handle">The <see cref="MemoryHandle{T}"/> to consume</param> + /// <param name="start">The offset from the begining of the buffer marking the begining of the string</param> + /// <param name="length">The number of characters this string points to</param> + /// <returns>The new <see cref="VnString"/></returns> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public static VnString ConsumeHandle( + MemoryHandle<char> handle, + +#if TARGET_64_BIT + ulong start, +#else + int start, +#endif + + int length) + { + if(length < 0) + { + throw new ArgumentOutOfRangeException(nameof(length)); + } + if((uint)length > handle.Length) + { + throw new ArgumentOutOfRangeException(nameof(length)); + } + return new VnString(handle, start, length); + } + /// <summary> + /// Asynchronously reads data from the specified stream and uses the specified encoding + /// to decode the binary data to a new <see cref="VnString"/> heap character buffer. + /// </summary> + /// <param name="stream">The stream to read data from</param> + /// <param name="encoding">The encoding to use while decoding data</param> + /// <param name="heap">The <see cref="IUnmangedHeap"/> to allocate buffers from</param> + /// <param name="bufferSize">The size of the buffer to allocate</param> + /// <returns>The new <see cref="VnString"/> containing the data</returns> + /// <exception cref="IOException"></exception> + /// <exception cref="OutOfMemoryException"></exception> + public static async ValueTask<VnString> FromStreamAsync(Stream stream, Encoding encoding, IUnmangedHeap heap, int bufferSize) + { + _ = stream ?? throw new ArgumentNullException(nameof(stream)); + _ = encoding ?? throw new ArgumentNullException(nameof(encoding)); + _ = heap ?? throw new ArgumentNullException(nameof(heap)); + + //Make sure the stream is readable + if (!stream.CanRead) + { + throw new IOException("The input stream is not readable"); + } + //See if the stream is a vn memory stream + if (stream is VnMemoryStream vnms) + { + //Get the number of characters + int numChars = encoding.GetCharCount(vnms.AsSpan()); + //New handle + MemoryHandle<char> charBuffer = heap.Alloc<char>(numChars); + try + { + //Write characters to character buffer + _ = encoding.GetChars(vnms.AsSpan(), charBuffer); + //Consume the new handle + return ConsumeHandle(charBuffer, 0, numChars); + } + catch + { + //If an error occured, dispose the buffer + charBuffer.Dispose(); + throw; + } + } + else + { + //Create a new char bufer starting with the buffer size + MemoryHandle<char> charBuffer = heap.Alloc<char>(bufferSize); + //Rent a temp binary buffer + IMemoryOwner<byte> binBuffer = heap.DirectAlloc<byte>(bufferSize); + try + { + int length = 0; + do + { + //read async + int read = await stream.ReadAsync(binBuffer.Memory); + //guard + if (read <= 0) + { + break; + } + //calculate the number of characters + int numChars = encoding.GetCharCount(binBuffer.Memory.Span[..read]); + //Guard for overflow + if (((ulong)(numChars + length)) >= int.MaxValue) + { + throw new OverflowException(); + } + //Re-alloc buffer + charBuffer.ResizeIfSmaller(length + numChars); + //Decode and update position + _ = encoding.GetChars(binBuffer.Memory.Span[..read], charBuffer.Span.Slice(length, numChars)); + //Update char count + length += numChars; + } while (true); + return ConsumeHandle(charBuffer, 0, length); + } + catch + { + //Free the memory allocated + charBuffer.Dispose(); + //We still want the exception to be propagated! + throw; + } + finally + { + //Dispose the binary buffer + binBuffer.Dispose(); + } + } + } + + /// <summary> + /// Gets the value of the character at the specified index + /// </summary> + /// <param name="index">The index of the character to get</param> + /// <returns>The <see cref="char"/> at the specified index within the buffer</returns> + /// <exception cref="ObjectDisposedException"></exception> + public char CharAt(int index) + { + //Check + Check(); + //Check bounds + return _stringSequence.Span[index]; + } + +#pragma warning disable IDE0057 // Use range operator + /// <summary> + /// Creates a <see cref="VnString"/> that is a window within the current string, + /// the referrence points to the same memory as the first instnace. + /// </summary> + /// <param name="start">The index within the current string to begin the child string</param> + /// <param name="count">The number of characters (or length) of the child string</param> + /// <returns>The child <see cref="VnString"/></returns> + /// <remarks> + /// Making substrings will reference the parents's underlying <see cref="MemoryHandle{T}"/> + /// and all children will be set in a disposed state when the parent instance is disposed + /// </remarks> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public VnString Substring(int start, int count) + { + //Check + Check(); + //Check bounds + if (start < 0 || (start + count) >= Length) + { + throw new ArgumentOutOfRangeException(nameof(count)); + } + //get sub-sequence slice for the current string + SubSequence<char> sub = _stringSequence.Slice((uint)start, count); + //Create new string with offsets pointing to same internal referrence + return new VnString(sub); + } + /// <summary> + /// Creates a <see cref="VnString"/> that is a window within the current string, + /// the referrence points to the same memory as the first instnace. + /// </summary> + /// <param name="start">The index within the current string to begin the child string</param> + /// <returns>The child <see cref="VnString"/></returns> + /// <remarks> + /// Making substrings will reference the parents's underlying <see cref="MemoryHandle{T}"/> + /// and all children will be set in a disposed state when the parent instance is disposed + /// </remarks> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public VnString Substring(int start) => Substring(start, (Length - start)); + public VnString this[Range range] + { + get + { + //get start + int start = range.Start.IsFromEnd ? Length - range.Start.Value : range.Start.Value; + //Get end + int end = range.End.IsFromEnd ? Length - range.End.Value : range.End.Value; + //Handle strings with no ending range + return (end >= start) ? Substring(start, (end - start)) : Substring(start); + } + } +#pragma warning restore IDE0057 // Use range operator + + /// <summary> + /// Gets a <see cref="ReadOnlySpan{T}"/> over the internal character buffer + /// </summary> + /// <returns></returns> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public ReadOnlySpan<char> AsSpan() + { + //Check + Check(); + return _stringSequence.Span; + } + + /// <summary> + /// Gets a <see cref="string"/> copy of the internal buffer + /// </summary> + /// <returns><see cref="string"/> representation of internal data</returns> + /// <exception cref="ObjectDisposedException"></exception> + public override unsafe string ToString() + { + //Check + Check(); + //Create a new + return AsSpan().ToString(); + } + /// <summary> + /// Gets the value of the character at the specified index + /// </summary> + /// <param name="index">The index of the character to get</param> + /// <returns>The <see cref="char"/> at the specified index within the buffer</returns> + /// <exception cref="ObjectDisposedException"></exception> + public char this[int index] => CharAt(index); + + //Casting to a vnstring should be explicit so the caller doesnt misuse memory managment + public static explicit operator ReadOnlySpan<char>(VnString? value) => Unsafe.IsNullRef(ref value) || value!.Disposed ? ReadOnlySpan<char>.Empty : value.AsSpan(); + public static explicit operator VnString(string value) => new (value); + public static explicit operator VnString(ReadOnlySpan<char> value) => new (value); + public static explicit operator VnString(char[] value) => new (value); + ///<inheritdoc/> + public override bool Equals(object? obj) + { + if(obj == null) + { + return false; + } + return obj switch + { + VnString => Equals(obj as VnString), //Use operator overload + string => Equals(obj as string), //Use operator overload + char[] => Equals(obj as char[]), //Use operator overload + _ => false, + }; + } + ///<inheritdoc/> + public bool Equals(VnString? other) => !ReferenceEquals(other, null) && Equals(other.AsSpan()); + ///<inheritdoc/> + public bool Equals(VnString? other, StringComparison stringComparison) => !ReferenceEquals(other, null) && Equals(other.AsSpan(), stringComparison); + ///<inheritdoc/> + public bool Equals(string? other) => Equals(other.AsSpan()); + ///<inheritdoc/> + public bool Equals(string? other, StringComparison stringComparison) => Equals(other.AsSpan(), stringComparison); + ///<inheritdoc/> + public bool Equals(char[]? other) => Equals(other.AsSpan()); + ///<inheritdoc/> + public bool Equals(char[]? other, StringComparison stringComparison) => Equals(other.AsSpan(), stringComparison); + ///<inheritdoc/> + public bool Equals(ReadOnlySpan<char> other, StringComparison stringComparison = StringComparison.Ordinal) => Length == other.Length && AsSpan().Equals(other, stringComparison); + ///<inheritdoc/> + public bool Equals(SubSequence<char> other) => Length == other.Size && AsSpan().SequenceEqual(other.Span); + ///<inheritdoc/> + public int CompareTo(string? other) => AsSpan().CompareTo(other, StringComparison.Ordinal); + ///<inheritdoc/> + ///<exception cref="ArgumentNullException"></exception> + public int CompareTo(VnString? other) + { + _ = other ?? throw new ArgumentNullException(nameof(other)); + return AsSpan().CompareTo(other.AsSpan(), StringComparison.Ordinal); + } + + /// <summary> + /// Gets a hashcode for the underyling string by using the .NET <see cref="string.GetHashCode()"/> + /// method on the character representation of the data + /// </summary> + /// <returns></returns> + /// <remarks> + /// It is safe to compare hashcodes of <see cref="VnString"/> to the <see cref="string"/> class or + /// a character span etc + /// </remarks> + /// <exception cref="ObjectDisposedException"></exception> + public override int GetHashCode() => string.GetHashCode(AsSpan()); + ///<inheritdoc/> + protected override void Free() + { + //Dispose the handle if we own it (null if we do not have the parent handle) + Handle?.Dispose(); + } + + public static bool operator ==(VnString left, VnString right) => ReferenceEquals(left, null) ? ReferenceEquals(right, null) : left.Equals(right); + + public static bool operator !=(VnString left, VnString right) => !(left == right); + + public static bool operator <(VnString left, VnString right) => ReferenceEquals(left, null) ? !ReferenceEquals(right, null) : left.CompareTo(right) < 0; + + public static bool operator <=(VnString left, VnString right) => ReferenceEquals(left, null) || left.CompareTo(right) <= 0; + + public static bool operator >(VnString left, VnString right) => !ReferenceEquals(left, null) && left.CompareTo(right) > 0; + + public static bool operator >=(VnString left, VnString right) => ReferenceEquals(left, null) ? ReferenceEquals(right, null) : left.CompareTo(right) >= 0; + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/VnTable.cs b/Utils/src/Memory/VnTable.cs new file mode 100644 index 0000000..1d5c0a6 --- /dev/null +++ b/Utils/src/Memory/VnTable.cs @@ -0,0 +1,213 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnTable.cs +* +* VnTable.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// Provides a Row-Major ordered table for use of storing value-types in umnaged heap memory + /// </summary> + /// <typeparam name="T"></typeparam> + public sealed class VnTable<T> : VnDisposeable, IIndexable<uint, T> where T : unmanaged + { + private readonly MemoryHandle<T>? BufferHandle; + /// <summary> + /// A value that indicates if the table does not contain any values + /// </summary> + public bool Empty { get; } + /// <summary> + /// The number of rows in the table + /// </summary> + public int Rows { get; } + /// <summary> + /// The nuber of columns in the table + /// </summary> + public int Cols { get; } + /// <summary> + /// Creates a new 2 dimensional table in unmanaged heap memory, using the <see cref="Memory.Shared"/> heap. + /// User should dispose of the table when no longer in use + /// </summary> + /// <param name="rows">Number of rows in the table</param> + /// <param name="cols">Number of columns in the table</param> + public VnTable(int rows, int cols) : this(Memory.Shared, rows, cols) { } + /// <summary> + /// Creates a new 2 dimensional table in unmanaged heap memory, using the specified heap. + /// User should dispose of the table when no longer in use + /// </summary> + /// <param name="heap"><see cref="PrivateHeap"/> to allocate table memory from</param> + /// <param name="rows">Number of rows in the table</param> + /// <param name="cols">Number of columns in the table</param> + public VnTable(IUnmangedHeap heap, int rows, int cols) + { + if (rows < 0 || cols < 0) + { + throw new ArgumentOutOfRangeException(nameof(rows), "Row and coulmn number must be 0 or larger"); + } + //empty table + if (rows == 0 && cols == 0) + { + Empty = true; + return; + } + + _ = heap ?? throw new ArgumentNullException(nameof(heap)); + + this.Rows = rows; + this.Cols = cols; + + long tableSize = Math.BigMul(rows, cols); + + //Alloc a buffer with zero memory enabled, with Rows * Cols number of elements + BufferHandle = heap.Alloc<T>(tableSize, true); + } + /// <summary> + /// Gets the value of an item in the table at the given indexes + /// </summary> + /// <param name="row">Row address of the item</param> + /// <param name="col">Column address of item</param> + /// <returns>The value of the item</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public T Get(int row, int col) + { + Check(); + if (this.Empty) + { + throw new InvalidOperationException("Table is empty"); + } + if (row < 0 || col < 0) + { + throw new ArgumentOutOfRangeException(nameof(row), "Row or column address less than 0"); + } + if (row > this.Rows) + { + throw new ArgumentOutOfRangeException(nameof(row), "Row out of range of current table"); + } + if (col > this.Cols) + { + throw new ArgumentOutOfRangeException(nameof(col), "Column address out of range of current table"); + } + //Calculate the address in memory for the item + //Calc row offset + long address = Cols * row; + //Calc column offset + address += col; + unsafe + { + //Get the value item + return *(BufferHandle!.GetOffset(address)); + } + } + /// <summary> + /// Sets the value of an item in the table at the given address + /// </summary> + /// <param name="item">Value of item to store</param> + /// <param name="row">Row address of the item</param> + /// <param name="col">Column address of item</param> + /// <returns>The value of the item</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public void Set(int row, int col, T item) + { + Check(); + if (this.Empty) + { + throw new InvalidOperationException("Table is empty"); + } + if (row < 0 || col < 0) + { + throw new ArgumentOutOfRangeException(nameof(row), "Row or column address less than 0"); + } + if (row > this.Rows) + { + throw new ArgumentOutOfRangeException(nameof(row), "Row out of range of current table"); + } + if (col > this.Cols) + { + throw new ArgumentOutOfRangeException(nameof(col), "Column address out of range of current table"); + } + //Calculate the address in memory for the item + //Calc row offset + long address = Cols * row; + //Calc column offset + address += col; + //Set the value item + unsafe + { + *BufferHandle!.GetOffset(address) = item; + } + } + /// <summary> + /// Equivalent to <see cref="VnTable{T}.Get(int, int)"/> and <see cref="VnTable{T}.Set(int, int, T)"/> + /// </summary> + /// <param name="row">Row address of item</param> + /// <param name="col">Column address of item</param> + /// <returns>The value of the item</returns> + public T this[int row, int col] + { + get => Get(row, col); + set => Set(row, col, value); + } + /// <summary> + /// Allows for direct addressing in the table. + /// </summary> + /// <param name="index"></param> + /// <returns></returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public unsafe T this[uint index] + { + get + { + Check(); + return !Empty ? *(BufferHandle!.GetOffset(index)) : throw new InvalidOperationException("Cannot index an empty table"); + } + + set + { + Check(); + if (Empty) + { + throw new InvalidOperationException("Cannot index an empty table"); + } + *(BufferHandle!.GetOffset(index)) = value; + } + } + ///<inheritdoc/> + protected override void Free() + { + if (!this.Empty) + { + //Dispose the buffer + BufferHandle!.Dispose(); + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/Memory/VnTempBuffer.cs b/Utils/src/Memory/VnTempBuffer.cs new file mode 100644 index 0000000..7726fe1 --- /dev/null +++ b/Utils/src/Memory/VnTempBuffer.cs @@ -0,0 +1,207 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnTempBuffer.cs +* +* VnTempBuffer.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using VNLib.Utils.Extensions; +using System.Security.Cryptography; + +namespace VNLib.Utils.Memory +{ + /// <summary> + /// A disposable temporary buffer from shared ArrayPool + /// </summary> + /// <typeparam name="T">Type of buffer to create</typeparam> + public sealed class VnTempBuffer<T> : VnDisposeable, IIndexable<int, T>, IMemoryHandle<T> + { + private readonly ArrayPool<T> Pool; + + /// <summary> + /// Referrence to internal buffer + /// </summary> + public T[] Buffer { get; private set; } + /// <summary> + /// Inital/desired size of internal buffer + /// </summary> + public int InitSize { get; } + + /// <summary> + /// Actual length of internal buffer + /// </summary> + public ulong Length => (ulong)Buffer.LongLength; + + /// <summary> + /// Actual length of internal buffer + /// </summary> + public int IntLength => Buffer.Length; + + ///<inheritdoc/> + ///<exception cref="ObjectDisposedException"></exception> + public Span<T> Span + { + get + { + Check(); + return new Span<T>(Buffer, 0, InitSize); + } + } + + /// <summary> + /// Allocates a new <see cref="VnTempBuffer{BufType}"/> with a new buffer from shared array-pool + /// </summary> + /// <param name="minSize">Minimum size of the buffer</param> + /// <param name="zero">Set the zero memory flag on close</param> + public VnTempBuffer(int minSize, bool zero = false) :this(ArrayPool<T>.Shared, minSize, zero) + {} + /// <summary> + /// Allocates a new <see cref="VnTempBuffer{BufType}"/> with a new buffer from specified array-pool + /// </summary> + /// <param name="pool">The <see cref="ArrayPool{T}"/> to allocate from and return to</param> + /// <param name="minSize">Minimum size of the buffer</param> + /// <param name="zero">Set the zero memory flag on close</param> + public VnTempBuffer(ArrayPool<T> pool, int minSize, bool zero = false) + { + Pool = pool; + Buffer = pool.Rent(minSize, zero); + InitSize = minSize; + } + /// <summary> + /// Gets an offset wrapper around the current buffer + /// </summary> + /// <param name="offset">Offset from begining of current buffer</param> + /// <param name="count">Number of <typeparamref name="T"/> from offset</param> + /// <returns>An <see cref="ArraySegment{BufType}"/> wrapper around the current buffer containing the offset</returns> + public ArraySegment<T> GetOffsetWrapper(int offset, int count) + { + Check(); + //Let arraysegment throw exceptions for checks + return new ArraySegment<T>(Buffer, offset, count); + } + ///<inheritdoc/> + public T this[int index] + { + get + { + Check(); + return Buffer[index]; + } + set + { + Check(); + Buffer[index] = value; + } + } + + /// <summary> + /// Gets a memory structure around the internal buffer + /// </summary> + /// <returns>A memory structure over the buffer</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public Memory<T> AsMemory() + { + Check(); + return new Memory<T>(Buffer, 0, InitSize); + } + /// <summary> + /// Gets a memory structure around the internal buffer + /// </summary> + /// <param name="count">The number of elements included in the result</param> + /// <param name="start">A value specifying the begining index of the buffer to include</param> + /// <returns>A memory structure over the buffer</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public Memory<T> AsMemory(int start, int count) + { + Check(); + return new Memory<T>(Buffer, start, count); + } + /// <summary> + /// Gets a memory structure around the internal buffer + /// </summary> + /// <param name="count">The number of elements included in the result</param> + /// <returns>A memory structure over the buffer</returns> + /// <exception cref="ObjectDisposedException"></exception> + /// <exception cref="ArgumentOutOfRangeException"></exception> + public Memory<T> AsMemory(int count) + { + Check(); + return new Memory<T>(Buffer, 0, count); + } + + /* + * Allow implict casts to span/arrayseg/memory + */ + public static implicit operator Memory<T>(VnTempBuffer<T> buf) => buf == null ? Memory<T>.Empty : buf.ToMemory(); + public static implicit operator Span<T>(VnTempBuffer<T> buf) => buf == null ? Span<T>.Empty : buf.ToSpan(); + public static implicit operator ArraySegment<T>(VnTempBuffer<T> buf) => buf == null ? ArraySegment<T>.Empty : buf.ToArraySegment(); + + public Memory<T> ToMemory() => Disposed ? Memory<T>.Empty : Buffer.AsMemory(0, InitSize); + public Span<T> ToSpan() => Disposed ? Span<T>.Empty : Buffer.AsSpan(0, InitSize); + public ArraySegment<T> ToArraySegment() => Disposed ? ArraySegment<T>.Empty : new(Buffer, 0, InitSize); + + /// <summary> + /// Returns buffer to shared array-pool + /// </summary> + protected override void Free() + { + //Return the buffer to the array pool + Pool.Return(Buffer); + + //Set buffer to null, +#pragma warning disable CS8625 // Cannot convert null literal to non-nullable reference type. + Buffer = null; +#pragma warning restore CS8625 // Cannot convert null literal to non-nullable reference type. + } + + unsafe MemoryHandle IPinnable.Pin(int elementIndex) + { + //Guard + if (elementIndex < 0 || elementIndex >= IntLength) + { + throw new ArgumentOutOfRangeException(nameof(elementIndex)); + } + + //Pin the array + GCHandle arrHandle = GCHandle.Alloc(Buffer, GCHandleType.Pinned); + //Get array base address + void* basePtr = (void*)arrHandle.AddrOfPinnedObject(); + //Get element offset + void* indexOffet = Unsafe.Add<T>(basePtr, elementIndex); + return new(indexOffet, arrHandle, this); + } + + void IPinnable.Unpin() + { + //Gchandle will manage the unpin + } + + ~VnTempBuffer() => Free(); + + + } +}
\ No newline at end of file diff --git a/Utils/src/Native/SafeLibraryHandle.cs b/Utils/src/Native/SafeLibraryHandle.cs new file mode 100644 index 0000000..4772bd4 --- /dev/null +++ b/Utils/src/Native/SafeLibraryHandle.cs @@ -0,0 +1,220 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SafeLibraryHandle.cs +* +* SafeLibraryHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Diagnostics; +using System.Runtime.InteropServices; +using System.Diagnostics.CodeAnalysis; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Native +{ + /// <summary> + /// Represents a safe handle to a native library loaded to the current process + /// </summary> + public sealed class SafeLibraryHandle : SafeHandle + { + ///<inheritdoc/> + public override bool IsInvalid => handle == IntPtr.Zero; + + private SafeLibraryHandle(IntPtr libHandle) : base(IntPtr.Zero, true) + { + //Init handle + SetHandle(libHandle); + } + + /// <summary> + /// Finds and loads the specified native libary into the current process by its name at runtime + /// </summary> + /// <param name="libPath">The path (or name of libary) to search for</param> + /// <param name="searchPath"> + /// The <see cref="DllImportSearchPath"/> used to search for libaries + /// within the current filesystem + /// </param> + /// <returns>The loaded <see cref="SafeLibraryHandle"/></returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="DllNotFoundException"></exception> + public static SafeLibraryHandle LoadLibrary(string libPath, DllImportSearchPath searchPath = DllImportSearchPath.ApplicationDirectory) + { + _ = libPath ?? throw new ArgumentNullException(nameof(libPath)); + //See if the path includes a file extension + return TryLoadLibrary(libPath, searchPath, out SafeLibraryHandle? lib) + ? lib + : throw new DllNotFoundException($"The library {libPath} or one of its dependencies could not be found"); + } + + /// <summary> + /// Attempts to load the specified native libary into the current process by its name at runtime + /// </summary> + ///<param name="libPath">The path (or name of libary) to search for</param> + /// <param name="searchPath"> + /// The <see cref="DllImportSearchPath"/> used to search for libaries + /// within the current filesystem + /// </param> + /// <param name="lib">The handle to the libary if successfully loaded</param> + /// <returns>True if the libary was found and loaded into the current process</returns> + public static bool TryLoadLibrary(string libPath, DllImportSearchPath searchPath, [NotNullWhen(true)] out SafeLibraryHandle? lib) + { + lib = null; + //Allow full rooted paths + if (Path.IsPathRooted(libPath)) + { + //Attempt a native load + if (NativeLibrary.TryLoad(libPath, out IntPtr libHandle)) + { + lib = new(libHandle); + return true; + } + return false; + } + //Check application directory first (including subdirectories) + if ((searchPath & DllImportSearchPath.ApplicationDirectory) > 0) + { + //get the current directory + string libDir = Directory.GetCurrentDirectory(); + if (TryLoadLibraryInternal(libDir, libPath, SearchOption.TopDirectoryOnly, out lib)) + { + return true; + } + } + //See if search in the calling assembly directory + if ((searchPath & DllImportSearchPath.AssemblyDirectory) > 0) + { + //Get the calling assmblies directory + string libDir = Assembly.GetCallingAssembly().Location; + Debug.WriteLine("Native library searching for calling assembly location:{0} ", libDir); + if (TryLoadLibraryInternal(libDir, libPath, SearchOption.TopDirectoryOnly, out lib)) + { + return true; + } + } + //Search system32 dir + if ((searchPath & DllImportSearchPath.System32) > 0) + { + //Get the system directory + string libDir = Environment.GetFolderPath(Environment.SpecialFolder.SystemX86); + if (TryLoadLibraryInternal(libDir, libPath, SearchOption.TopDirectoryOnly, out lib)) + { + return true; + } + } + //Attempt a native load + { + if (NativeLibrary.TryLoad(libPath, out IntPtr libHandle)) + { + lib = new(libHandle); + return true; + } + return false; + } + } + + private static bool TryLoadLibraryInternal(string libDir, string libPath, SearchOption dirSearchOptions, [NotNullWhen(true)] out SafeLibraryHandle? libary) + { + //Try to find the libary file + string? libFile = GetLibraryFile(libDir, libPath, dirSearchOptions); + //Load libary + if (libFile != null && NativeLibrary.TryLoad(libFile, out IntPtr libHandle)) + { + libary = new SafeLibraryHandle(libHandle); + return true; + } + libary = null; + return false; + } + private static string? GetLibraryFile(string dirPath, string libPath, SearchOption search) + { + //slice the lib to its file name + libPath = Path.GetFileName(libPath); + libPath = Path.ChangeExtension(libPath, OperatingSystem.IsWindows() ? ".dll" : ".so"); + //Select the first file that matches the name + return Directory.EnumerateFiles(dirPath, libPath, search).FirstOrDefault(); + } + + /// <summary> + /// Loads a native method from the library of the specified name and managed delegate + /// </summary> + /// <typeparam name="T">The native method delegate type</typeparam> + /// <param name="methodName">The name of the native method</param> + /// <returns>A wapper handle around the native method delegate</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException">If the handle is closed or invalid</exception> + /// <exception cref="EntryPointNotFoundException">When the specified entrypoint could not be found</exception> + public SafeMethodHandle<T> GetMethod<T>(string methodName) where T : Delegate + { + //Increment handle count before obtaining a method + bool success = false; + DangerousAddRef(ref success); + if (!success) + { + throw new ObjectDisposedException("The libary has been released!"); + } + try + { + //Get the method pointer + IntPtr nativeMethod = NativeLibrary.GetExport(handle, methodName); + //Get the delegate for the function pointer + T method = Marshal.GetDelegateForFunctionPointer<T>(nativeMethod); + return new(this, method); + } + catch + { + DangerousRelease(); + throw; + } + } + /// <summary> + /// Gets an delegate wrapper for the specified method without tracking its referrence. + /// The caller must manage the <see cref="SafeLibraryHandle"/> referrence count in order + /// to not leak resources or cause process corruption + /// </summary> + /// <typeparam name="T">The native method delegate type</typeparam> + /// <param name="methodName">The name of the native method</param> + /// <returns>A the delegate wrapper on the native method</returns> + /// <exception cref="ArgumentNullException"></exception> + /// <exception cref="ObjectDisposedException">If the handle is closed or invalid</exception> + /// <exception cref="EntryPointNotFoundException">When the specified entrypoint could not be found</exception> + public T DangerousGetMethod<T>(string methodName) where T : Delegate + { + this.ThrowIfClosed(); + //Get the method pointer + IntPtr nativeMethod = NativeLibrary.GetExport(handle, methodName); + //Get the delegate for the function pointer + return Marshal.GetDelegateForFunctionPointer<T>(nativeMethod); + } + + ///<inheritdoc/> + protected override bool ReleaseHandle() + { + //Free the library and set the handle as invalid + NativeLibrary.Free(handle); + SetHandleAsInvalid(); + return true; + } + } +} diff --git a/Utils/src/Native/SafeMethodHandle.cs b/Utils/src/Native/SafeMethodHandle.cs new file mode 100644 index 0000000..3ba0879 --- /dev/null +++ b/Utils/src/Native/SafeMethodHandle.cs @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: SafeMethodHandle.cs +* +* SafeMethodHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +using VNLib.Utils.Resources; + +namespace VNLib.Utils.Native +{ + /// <summary> + /// Represents a handle to a <see cref="SafeLibraryHandle"/>'s + /// native method + /// </summary> + /// <typeparam name="T">The native method deelgate type</typeparam> + public class SafeMethodHandle<T> : OpenHandle where T : Delegate + { + private T? _method; + private readonly SafeLibraryHandle Library; + + internal SafeMethodHandle(SafeLibraryHandle lib, T method) + { + Library = lib; + _method = method; + } + + /// <summary> + /// A delegate to the native method + /// </summary> + public T? Method => _method; + + ///<inheritdoc/> + protected override void Free() + { + //Release the method + _method = default; + //Decrement lib handle count + Library.DangerousRelease(); + } + } +} diff --git a/Utils/src/NativeLibraryException.cs b/Utils/src/NativeLibraryException.cs new file mode 100644 index 0000000..5c66852 --- /dev/null +++ b/Utils/src/NativeLibraryException.cs @@ -0,0 +1,89 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: NativeLibraryException.cs +* +* NativeLibraryException.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils +{ + /// <summary> + /// Raised when an internal buffer was not propery sized for the opreation + /// </summary> + public class InternalBufferTooSmallException : OutOfMemoryException + { + public InternalBufferTooSmallException(string message) : base(message) + {} + + public InternalBufferTooSmallException(string message, Exception innerException) : base(message, innerException) + {} + + public InternalBufferTooSmallException() + {} + } + + /// <summary> + /// A base class for all native library related exceptions + /// </summary> + public class NativeLibraryException : SystemException + { + public NativeLibraryException(string message) : base(message) + {} + + public NativeLibraryException(string message, Exception innerException) : base(message, innerException) + {} + + public NativeLibraryException() + {} + } + + /// <summary> + /// Base exception class for native memory related exceptions + /// </summary> + public class NativeMemoryException : NativeLibraryException + { + public NativeMemoryException(string message) : base(message) + {} + + public NativeMemoryException(string message, Exception innerException) : base(message, innerException) + {} + + public NativeMemoryException() + {} + } + + /// <summary> + /// Raised when a memory allocation or resize failed because there is + /// no more memory available + /// </summary> + public class NativeMemoryOutOfMemoryException : OutOfMemoryException + { + public NativeMemoryOutOfMemoryException(string message) : base(message) + {} + + public NativeMemoryOutOfMemoryException(string message, Exception innerException) : base(message, innerException) + {} + + public NativeMemoryOutOfMemoryException() + {} + } +} diff --git a/Utils/src/Resources/BackedResourceBase.cs b/Utils/src/Resources/BackedResourceBase.cs new file mode 100644 index 0000000..0a2e1e3 --- /dev/null +++ b/Utils/src/Resources/BackedResourceBase.cs @@ -0,0 +1,79 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: BackedResourceBase.cs +* +* BackedResourceBase.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Text.Json; +using System.Runtime.CompilerServices; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// A base class for a resource that is backed by an external data store. + /// Implements the <see cref="IResource"/> interfaceS + /// </summary> + public abstract class BackedResourceBase : IResource + { + ///<inheritdoc/> + public bool IsReleased { get; protected set; } + + /// <summary> + /// Optional <see cref="JsonSerializerOptions"/> to be used when serializing + /// the resource + /// </summary> + internal protected virtual JsonSerializerOptions? JSO { get; } + + /// <summary> + /// A value indicating whether the instance should be deleted when released + /// </summary> + protected bool Deleted { get; set; } + /// <summary> + /// A value indicating whether the instance should be updated when released + /// </summary> + protected bool Modified { get; set; } + + /// <summary> + /// Checks if the resouce has been disposed and raises an exception if it is + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected void Check() + { + if (IsReleased) + { + throw new ObjectDisposedException("The resource has been disposed"); + } + } + + /// <summary> + /// Returns the JSON serializable resource to be updated during an update + /// </summary> + /// <returns>The resource to update</returns> + protected abstract object GetResource(); + + /// <summary> + /// Marks the resource for deletion from backing store during closing events + /// </summary> + public virtual void Delete() => Deleted = true; + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/CallbackOpenHandle.cs b/Utils/src/Resources/CallbackOpenHandle.cs new file mode 100644 index 0000000..625bd45 --- /dev/null +++ b/Utils/src/Resources/CallbackOpenHandle.cs @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: CallbackOpenHandle.cs +* +* CallbackOpenHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// A concrete <see cref="OpenHandle"/> for a defered operation or a resource that should be released or unwound + /// when the instance lifetime has ended. + /// </summary> + public sealed class CallbackOpenHandle : OpenHandle + { + private readonly Action ReleaseFunc; + /// <summary> + /// Creates a new generic <see cref="OpenHandle"/> with the specified release callback method + /// </summary> + /// <param name="release">The callback function to invoke when the <see cref="OpenHandle"/> is disposed</param> + public CallbackOpenHandle(Action release) => ReleaseFunc = release; + ///<inheritdoc/> + protected override void Free() => ReleaseFunc(); + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/ExclusiveResourceHandle.cs b/Utils/src/Resources/ExclusiveResourceHandle.cs new file mode 100644 index 0000000..173bdd1 --- /dev/null +++ b/Utils/src/Resources/ExclusiveResourceHandle.cs @@ -0,0 +1,81 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ExclusiveResourceHandle.cs +* +* ExclusiveResourceHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// While in scope, holds an exclusive lock on the specified object that implements the <see cref="IExclusiveResource"/> interface + /// </summary> + /// <typeparam name="T"></typeparam> + public class ExclusiveResourceHandle<T> : OpenResourceHandle<T> where T : IExclusiveResource + { + private readonly Action Release; + private readonly Lazy<T> LazyVal; + + /// <summary> + /// <inheritdoc/> + /// <br></br> + /// <br></br> + /// This value is lazy inialized and will invoke the factory function on first access. + /// Accessing this variable is thread safe while the handle is in scope + /// <br></br> + /// <br></br> + /// Exceptions will be propagated during initialziation + /// </summary> + public override T Resource => LazyVal.Value; + + /// <summary> + /// Creates a new <see cref="ExclusiveResourceHandle{TResource}"/> wrapping the + /// <see cref="IExclusiveResource"/> object to manage its lifecycle and reuse + /// </summary> + /// <param name="factory">Factory function that will generate the value when used</param> + /// <param name="release">Callback function that will be invoked after object gets disposed</param> + internal ExclusiveResourceHandle(Func<T> factory, Action release) + { + //Store the release action + Release = release; + //Store the new lazy val from the factory function (enabled thread safey) + LazyVal = new(factory, System.Threading.LazyThreadSafetyMode.ExecutionAndPublication); + } + + protected override void Free() + { + try + { + //Dispose the value if it was created, otherwise do not create it + if (LazyVal.IsValueCreated) + { + Resource?.Release(); + } + } + finally + { + //Always invoke the release callback + Release(); + } + } + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/IExclusiveResource.cs b/Utils/src/Resources/IExclusiveResource.cs new file mode 100644 index 0000000..43ec607 --- /dev/null +++ b/Utils/src/Resources/IExclusiveResource.cs @@ -0,0 +1,39 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IExclusiveResource.cs +* +* IExclusiveResource.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Utils.Resources +{ + + /// <summary> + /// An object, that when used in a mulithreading context, guaruntees that the caller has exclusive + /// access to the instance and relinquishes exclusive access when Release() is called; + /// </summary> + public interface IExclusiveResource : IResource + { + /// <summary> + /// Releases the resource from use. Called when a <see cref="ExclusiveResourceHandle{T}"/> is disposed + /// </summary> + void Release(); + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/IResource.cs b/Utils/src/Resources/IResource.cs new file mode 100644 index 0000000..345e284 --- /dev/null +++ b/Utils/src/Resources/IResource.cs @@ -0,0 +1,38 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: IResource.cs +* +* IResource.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// The base interface for any resource that may be backed by an external + /// data store, and has a finite lifetime, usually accessed within a handle + /// </summary> + public interface IResource + { + /// <summary> + /// Gets a value indicating if the resource has been released + /// </summary> + bool IsReleased { get; } + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/OpenHandle.cs b/Utils/src/Resources/OpenHandle.cs new file mode 100644 index 0000000..6133a65 --- /dev/null +++ b/Utils/src/Resources/OpenHandle.cs @@ -0,0 +1,38 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: OpenHandle.cs +* +* OpenHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// Represents a base class for an open resource or operation that is valid while being held, + /// and is released or unwound when disposed. + /// </summary> + /// <remarks> + /// The <see cref="OpenHandle"/> pattern, may throw exceptions when disposed as deferred + /// release actions are completed + /// </remarks> + public abstract class OpenHandle : VnDisposeable + { + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/OpenResourceHandle.cs b/Utils/src/Resources/OpenResourceHandle.cs new file mode 100644 index 0000000..d9f9fd2 --- /dev/null +++ b/Utils/src/Resources/OpenResourceHandle.cs @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: OpenResourceHandle.cs +* +* OpenResourceHandle.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// <para> + /// An abstract base class for an <see cref="OpenHandle"/> that holds a specific resource and manages its lifetime. + /// </para> + /// </summary> + /// <inheritdoc/> + /// <typeparam name="TResource">The resource type</typeparam> + public abstract class OpenResourceHandle<TResource> : OpenHandle + { + /// <summary> + /// The resource held by the open handle + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + public abstract TResource Resource { get; } + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/ResourceDeleteFailedException.cs b/Utils/src/Resources/ResourceDeleteFailedException.cs new file mode 100644 index 0000000..8e796b5 --- /dev/null +++ b/Utils/src/Resources/ResourceDeleteFailedException.cs @@ -0,0 +1,40 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ResourceDeleteFailedException.cs +* +* ResourceDeleteFailedException.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// Raised when a resource delete has failed + /// </summary> + public class ResourceDeleteFailedException : Exception + { + public ResourceDeleteFailedException() { } + public ResourceDeleteFailedException(string message) : base(message) { } + public ResourceDeleteFailedException(string message, Exception innerException) : base(message, innerException) { } + protected ResourceDeleteFailedException(SerializationInfo info, StreamingContext context) : base(info, context) { } + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/ResourceUpdateFailedException.cs b/Utils/src/Resources/ResourceUpdateFailedException.cs new file mode 100644 index 0000000..b4b2b3a --- /dev/null +++ b/Utils/src/Resources/ResourceUpdateFailedException.cs @@ -0,0 +1,40 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: ResourceUpdateFailedException.cs +* +* ResourceUpdateFailedException.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.Serialization; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// Raised when a resource update has failed + /// </summary> + public class ResourceUpdateFailedException : Exception + { + public ResourceUpdateFailedException() { } + public ResourceUpdateFailedException(string message) : base(message) { } + public ResourceUpdateFailedException(string message, Exception innerException) : base(message, innerException) { } + protected ResourceUpdateFailedException(SerializationInfo info, StreamingContext context) : base(info, context) { } + } +}
\ No newline at end of file diff --git a/Utils/src/Resources/UpdatableResource.cs b/Utils/src/Resources/UpdatableResource.cs new file mode 100644 index 0000000..16f26f2 --- /dev/null +++ b/Utils/src/Resources/UpdatableResource.cs @@ -0,0 +1,113 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: UpdatableResource.cs +* +* UpdatableResource.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; + +using VNLib.Utils.IO; + +namespace VNLib.Utils.Resources +{ + /// <summary> + /// A callback delegate used for updating a <see cref="UpdatableResource"/> + /// </summary> + /// <param name="source">The <see cref="UpdatableResource"/> to be updated</param> + /// <param name="data">The serialized data to be stored/updated</param> + /// <exception cref="ResourceUpdateFailedException"></exception> + public delegate void UpdateCallback(object source, Stream data); + /// <summary> + /// A callback delegate invoked when a <see cref="UpdatableResource"/> delete is requested + /// </summary> + /// <param name="source">The <see cref="UpdatableResource"/> to be deleted</param> + /// <exception cref="ResourceDeleteFailedException"></exception> + public delegate void DeleteCallback(object source); + + /// <summary> + /// Implemented by a resource that is backed by an external data store, that when modified or deleted will + /// be reflected to the backing store. + /// </summary> + public abstract class UpdatableResource : BackedResourceBase, IExclusiveResource + { + /// <summary> + /// The update callback method to invoke during a release operation + /// when the resource is updated. + /// </summary> + protected abstract UpdateCallback UpdateCb { get; } + /// <summary> + /// The callback method to invoke during a realease operation + /// when the resource should be deleted + /// </summary> + protected abstract DeleteCallback DeleteCb { get; } + + /// <summary> + /// <inheritdoc/> + /// </summary> + /// <exception cref="InvalidOperationException"></exception> + /// <exception cref="ResourceDeleteFailedException"></exception> + /// <exception cref="ResourceUpdateFailedException"></exception> + public virtual void Release() + { + //If resource has already been realeased, return + if (IsReleased) + { + return; + } + //If deleted flag is set, invoke the delete callback + if (Deleted) + { + DeleteCb(this); + } + //If the state has been modifed, flush changes to the store + else if (Modified) + { + FlushPendingChanges(); + } + //Set the released value + IsReleased = true; + } + + /// <summary> + /// Writes the current state of the the resource to the backing store + /// immediatly by invoking the specified callback. + /// <br></br> + /// <br></br> + /// Only call this method if your store supports multiple state updates + /// </summary> + protected virtual void FlushPendingChanges() + { + //Get the resource + object resource = GetResource(); + //Open a memory stream to store data in + using VnMemoryStream data = new(); + //Serialize and write to stream + VnEncoding.JSONSerializeToBinary(resource, data, resource.GetType(), JSO); + //Reset stream to begining + _ = data.Seek(0, SeekOrigin.Begin); + //Invoke update callback + UpdateCb(this, data); + //Clear modified flag + Modified = false; + } + } +}
\ No newline at end of file diff --git a/Utils/src/VNLib.Utils.csproj b/Utils/src/VNLib.Utils.csproj new file mode 100644 index 0000000..b14ab27 --- /dev/null +++ b/Utils/src/VNLib.Utils.csproj @@ -0,0 +1,47 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <RootNamespace>VNLib.Utils</RootNamespace> + <Authors>Vaughn Nugent</Authors> + <Product>VNLib Utilities Library</Product> + <Copyright>Copyright © 2022 Vaughn Nugent</Copyright> + <PackageProjectUrl>https://www.vaughnnugent.com/resources</PackageProjectUrl> + <AssemblyName>VNLib.Utils</AssemblyName> + <Version>1.0.1.10</Version> + <Description>Base utilities library, structs, classes</Description> + <AllowUnsafeBlocks>true</AllowUnsafeBlocks> + <Nullable>enable</Nullable> + <GenerateDocumentationFile>True</GenerateDocumentationFile> + <AnalysisLevel>latest-all</AnalysisLevel> + <SignAssembly>True</SignAssembly> + <AssemblyOriginatorKeyFile>\\vaughnnugent.com\Internal\Folder Redirection\vman\Documents\Programming\Software\StrongNameingKey.snk</AssemblyOriginatorKeyFile> + </PropertyGroup> + + <ItemGroup> + <None Include="..\.editorconfig" Link=".editorconfig" /> + </ItemGroup> + + <PropertyGroup Condition="'$(Configuration)'=='Debug'"> + <CheckForOverflowUnderflow>true</CheckForOverflowUnderflow> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'"> + <Deterministic>False</Deterministic> + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> +</Project> diff --git a/Utils/src/VnDisposeable.cs b/Utils/src/VnDisposeable.cs new file mode 100644 index 0000000..4230a13 --- /dev/null +++ b/Utils/src/VnDisposeable.cs @@ -0,0 +1,85 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnDisposeable.cs +* +* VnDisposeable.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Runtime.CompilerServices; + +namespace VNLib.Utils +{ + /// <summary> + /// Provides a base class with abstract methods for for disposable objects, with disposed check method + /// </summary> + public abstract class VnDisposeable : IDisposable + { + ///<inheritdoc/> + protected bool Disposed { get; private set; } + + /// <summary> + /// When overriden in a child class, is responsible for freeing resources + /// </summary> + protected abstract void Free(); + + /// <summary> + /// Checks if the current object has been disposed. Method will be inlined where possible + /// </summary> + /// <exception cref="ObjectDisposedException"></exception> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected virtual void Check() + { + if (Disposed) + { + throw new ObjectDisposedException("Object has been disposed"); + } + } + + /// <summary> + /// Sets the internal state to diposed without calling <see cref="Free"/> operation. + /// Usefull if another code-path performs the free operation independant of a dispose opreation. + /// </summary> + [MethodImpl(MethodImplOptions.AggressiveInlining)] + protected void SetDisposedState() => Disposed = true; + ///<inheritdoc/> + protected virtual void Dispose(bool disposing) + { + if (!Disposed) + { + if (disposing) + { + //Call free method + Free(); + } + Disposed = true; + } + } + //Finalizer is not needed here + + ///<inheritdoc/> + public void Dispose() + { + // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method + Dispose(disposing: true); + GC.SuppressFinalize(this); + } + } +}
\ No newline at end of file diff --git a/Utils/src/VnEncoding.cs b/Utils/src/VnEncoding.cs new file mode 100644 index 0000000..94d8a1a --- /dev/null +++ b/Utils/src/VnEncoding.cs @@ -0,0 +1,914 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.Utils +* File: VnEncoding.cs +* +* VnEncoding.cs is part of VNLib.Utils which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.Utils is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.Utils is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.Utils. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.IO; +using System.Text; +using System.Buffers; +using System.Text.Json; +using System.Threading; +using System.Buffers.Text; +using System.Threading.Tasks; +using System.Runtime.InteropServices; +using System.Runtime.CompilerServices; + +using VNLib.Utils.IO; +using VNLib.Utils.Memory; +using VNLib.Utils.Extensions; + + +namespace VNLib.Utils +{ + /// <summary> + /// Contains static methods for encoding data + /// </summary> + public static class VnEncoding + { + /// <summary> + /// Encodes a <see cref="ReadOnlySpan{T}"/> with the specified <see cref="Encoding"/> to a <see cref="VnMemoryStream"/> that must be disposed by the user + /// </summary> + /// <param name="data">Data to be encoded</param> + /// <param name="encoding"><see cref="Encoding"/> to encode data with</param> + /// <returns>A <see cref="Stream"/> contating the encoded data</returns> + public static VnMemoryStream GetMemoryStream(in ReadOnlySpan<char> data, Encoding encoding) + { + _ = encoding ?? throw new ArgumentNullException(nameof(encoding)); + //Create new memory handle to copy data to + MemoryHandle<byte>? handle = null; + try + { + //get number of bytes + int byteCount = encoding.GetByteCount(data); + //resize the handle to fit the data + handle = Memory.Memory.Shared.Alloc<byte>(byteCount); + //encode + int size = encoding.GetBytes(data, handle); + //Consume the handle into a new vnmemstream and return it + return VnMemoryStream.ConsumeHandle(handle, size, true); + } + catch + { + //Dispose the handle if there is an excpetion + handle?.Dispose(); + throw; + } + } + + /// <summary> + /// Attempts to deserialze a json object from a stream of UTF8 data + /// </summary> + /// <typeparam name="T">The type of the object to deserialize</typeparam> + /// <param name="data">Binary data to read from</param> + /// <param name="options"><see cref="JsonSerializerOptions"/> object to pass to deserializer</param> + /// <returns>The object decoded from the stream</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static T? JSONDeserializeFromBinary<T>(Stream? data, JsonSerializerOptions? options = null) + { + //Return default if null + if (data == null) + { + return default; + } + //Create a memory stream as a buffer + using VnMemoryStream ms = new(); + //Copy stream data to memory + data.CopyTo(ms, null); + if (ms.Length > 0) + { + //Rewind + ms.Position = 0; + //Recover data from stream + return ms.AsSpan().AsJsonObject<T>(options); + } + //Stream is empty + return default; + } + /// <summary> + /// Attempts to deserialze a json object from a stream of UTF8 data + /// </summary> + /// <param name="data">Binary data to read from</param> + /// <param name="type"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> object to pass to deserializer</param> + /// <returns>The object decoded from the stream</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static object? JSONDeserializeFromBinary(Stream? data, Type type, JsonSerializerOptions? options = null) + { + //Return default if null + if (data == null) + { + return default; + } + //Create a memory stream as a buffer + using VnMemoryStream ms = new(); + //Copy stream data to memory + data.CopyTo(ms, null); + if (ms.Length > 0) + { + //Rewind + ms.Position = 0; + //Recover data from stream + return JsonSerializer.Deserialize(ms.AsSpan(), type, options); + } + //Stream is empty + return default; + } + /// <summary> + /// Attempts to deserialze a json object from a stream of UTF8 data + /// </summary> + /// <typeparam name="T">The type of the object to deserialize</typeparam> + /// <param name="data">Binary data to read from</param> + /// <param name="options"><see cref="JsonSerializerOptions"/> object to pass to deserializer</param> + /// <param name="cancellationToken"></param> + /// <returns>The object decoded from the stream</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static ValueTask<T?> JSONDeserializeFromBinaryAsync<T>(Stream? data, JsonSerializerOptions? options = null, CancellationToken cancellationToken = default) + { + //Return default if null + return data == null || data.Length == 0 ? ValueTask.FromResult<T?>(default) : JsonSerializer.DeserializeAsync<T>(data, options, cancellationToken); + } + /// <summary> + /// Attempts to deserialze a json object from a stream of UTF8 data + /// </summary> + /// <param name="data">Binary data to read from</param> + /// <param name="type"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> object to pass to deserializer</param> + /// <param name="cancellationToken"></param> + /// <returns>The object decoded from the stream</returns> + /// <exception cref="JsonException"></exception> + /// <exception cref="NotSupportedException"></exception> + public static ValueTask<object?> JSONDeserializeFromBinaryAsync(Stream? data, Type type, JsonSerializerOptions? options = null, CancellationToken cancellationToken = default) + { + //Return default if null + return data == null || data.Length == 0 ? ValueTask.FromResult<object?>(default) : JsonSerializer.DeserializeAsync(data, type, options, cancellationToken); + } + /// <summary> + /// Attempts to serialize the object to json and write the encoded data to the stream + /// </summary> + /// <typeparam name="T">The object type to serialize</typeparam> + /// <param name="data">The object to serialize</param> + /// <param name="output">The <see cref="Stream"/> to write output data to</param> + /// <param name="options"><see cref="JsonSerializerOptions"/> object to pass to serializer</param> + /// <exception cref="JsonException"></exception> + public static void JSONSerializeToBinary<T>(T data, Stream output, JsonSerializerOptions? options = null) + { + //return if null + if(data == null) + { + return; + } + //Serialize + JsonSerializer.Serialize(output, data, options); + } + /// <summary> + /// Attempts to serialize the object to json and write the encoded data to the stream + /// </summary> + /// <param name="data">The object to serialize</param> + /// <param name="output">The <see cref="Stream"/> to write output data to</param> + /// <param name="type"></param> + /// <param name="options"><see cref="JsonSerializerOptions"/> object to pass to serializer</param> + /// <exception cref="JsonException"></exception> + public static void JSONSerializeToBinary(object data, Stream output, Type type, JsonSerializerOptions? options = null) + { + //return if null + if (data == null) + { + return; + } + //Serialize + JsonSerializer.Serialize(output, data, type, options); + } + + #region Base32 + + private const string RFC_4648_BASE32_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567"; + + /// <summary> + /// Attempts to convert the specified byte sequence in Base32 encoding + /// and writing the encoded data to the output buffer. + /// </summary> + /// <param name="input">The input buffer to convert</param> + /// <param name="output">The ouput buffer to write encoded data to</param> + /// <returns>The number of characters written, false if no data was written or output buffer was too small</returns> + public static ERRNO TryToBase32Chars(ReadOnlySpan<byte> input, Span<char> output) + { + ForwardOnlyWriter<char> writer = new(output); + return TryToBase32Chars(input, ref writer); + } + /// <summary> + /// Attempts to convert the specified byte sequence in Base32 encoding + /// and writing the encoded data to the output buffer. + /// </summary> + /// <param name="input">The input buffer to convert</param> + /// <param name="writer">A <see cref="ForwardOnlyWriter{T}"/> to write encoded chars to</param> + /// <returns>The number of characters written, false if no data was written or output buffer was too small</returns> + public static ERRNO TryToBase32Chars(ReadOnlySpan<byte> input, ref ForwardOnlyWriter<char> writer) + { + //calculate char size + int charCount = (int)Math.Ceiling(input.Length / 5d) * 8; + //Make sure there is enough room + if(charCount > writer.RemainingSize) + { + return false; + } + //sliding window over input buffer + ForwardOnlyReader<byte> reader = new(input); + + while (reader.WindowSize > 0) + { + //Convert the current window + WriteChars(reader.Window, ref writer); + //shift the window + reader.Advance(Math.Min(5, reader.WindowSize)); + } + return writer.Written; + } + private unsafe static void WriteChars(ReadOnlySpan<byte> input, ref ForwardOnlyWriter<char> writer) + { + //Get the input buffer as long + ulong inputAsLong = 0; + //Get a byte pointer over the ulong to index it as a byte buffer + byte* buffer = (byte*)&inputAsLong; + //Check proc endianness + if (BitConverter.IsLittleEndian) + { + //store each byte consecutivley and allow for padding + for (int i = 0; (i < 5 && i < input.Length); i++) + { + //Write bytes from upper to lower byte order for little endian systems + buffer[7 - i] = input[i]; + } + } + else + { + //store each byte consecutivley and allow for padding + for (int i = 0; (i < 5 && i < input.Length); i++) + { + //Write bytes from lower to upper byte order for Big Endian systems + buffer[i] = input[i]; + } + } + int rounds = (input.Length) switch + { + 1 => 2, + 2 => 4, + 3 => 5, + 4 => 7, + _ => 8 + }; + //Convert each byte segment up to the number of bytes encoded + for (int i = 0; i < rounds; i++) + { + //store the leading byte + byte val = buffer[7]; + //right shift the value to lower 5 bits + val >>= 3; + //Lookup charcode + char base32Char = RFC_4648_BASE32_CHARS[val]; + //append the character to the writer + writer.Append(base32Char); + //Shift input left by 5 bits so the next 5 bits can be read + inputAsLong <<= 5; + } + //Fill remaining bytes with padding chars + for(; rounds < 8; rounds++) + { + //Append trailing '=' + writer.Append('='); + } + } + + /// <summary> + /// Attempts to decode the Base32 encoded string + /// </summary> + /// <param name="input">The Base32 encoded data to decode</param> + /// <param name="output">The output buffer to write decoded data to</param> + /// <returns>The number of bytes written to the output</returns> + /// <exception cref="FormatException"></exception> + public static ERRNO TryFromBase32Chars(ReadOnlySpan<char> input, Span<byte> output) + { + ForwardOnlyWriter<byte> writer = new(output); + return TryFromBase32Chars(input, ref writer); + } + /// <summary> + /// Attempts to decode the Base32 encoded string + /// </summary> + /// <param name="input">The Base32 encoded data to decode</param> + /// <param name="writer">A <see cref="ForwardOnlyWriter{T}"/> to write decoded bytes to</param> + /// <returns>The number of bytes written to the output</returns> + /// <exception cref="FormatException"></exception> + public unsafe static ERRNO TryFromBase32Chars(ReadOnlySpan<char> input, ref ForwardOnlyWriter<byte> writer) + { + //TODO support Big-Endian byte order + + //trim padding characters + input = input.Trim('='); + //Calc the number of bytes to write + int outputSize = (input.Length * 5) / 8; + //make sure the output buffer is large enough + if(writer.RemainingSize < outputSize) + { + return false; + } + + //buffer used to shift data while decoding + ulong bufferLong = 0; + + //re-cast to byte* to index it as a byte buffer + byte* buffer = (byte*)&bufferLong; + + int count = 0, len = input.Length; + while(count < len) + { + //Convert the character to its char code + byte charCode = GetCharCode(input[count]); + //write byte to buffer + buffer[0] |= charCode; + count++; + //If 8 characters have been decoded, reset the buffer + if((count % 8) == 0) + { + //Write the 5 upper bytes in reverse order to the output buffer + for(int j = 0; j < 5; j++) + { + writer.Append(buffer[4 - j]); + } + //reset + bufferLong = 0; + } + //left shift the buffer up by 5 bits + bufferLong <<= 5; + } + //If remaining data has not be written, but has been buffed, finalize it + if (writer.Written < outputSize) + { + //calculate how many bits the buffer still needs to be shifted by (will be 5 bits off because of the previous loop) + int remainingShift = (7 - (count % 8)) * 5; + //right shift the buffer by the remaining bit count + bufferLong <<= remainingShift; + //calc remaining bytes + int remaining = (outputSize - writer.Written); + //Write remaining bytes to the output + for(int i = 0; i < remaining; i++) + { + writer.Append(buffer[4 - i]); + } + } + return writer.Written; + } + private static byte GetCharCode(char c) + { + //cast to byte to get its base 10 value + return c switch + { + //Upper case + 'A' => 0, + 'B' => 1, + 'C' => 2, + 'D' => 3, + 'E' => 4, + 'F' => 5, + 'G' => 6, + 'H' => 7, + 'I' => 8, + 'J' => 9, + 'K' => 10, + 'L' => 11, + 'M' => 12, + 'N' => 13, + 'O' => 14, + 'P' => 15, + 'Q' => 16, + 'R' => 17, + 'S' => 18, + 'T' => 19, + 'U' => 20, + 'V' => 21, + 'W' => 22, + 'X' => 23, + 'Y' => 24, + 'Z' => 25, + //Lower case + 'a' => 0, + 'b' => 1, + 'c' => 2, + 'd' => 3, + 'e' => 4, + 'f' => 5, + 'g' => 6, + 'h' => 7, + 'i' => 8, + 'j' => 9, + 'k' => 10, + 'l' => 11, + 'm' => 12, + 'n' => 13, + 'o' => 14, + 'p' => 15, + 'q' => 16, + 'r' => 17, + 's' => 18, + 't' => 19, + 'u' => 20, + 'v' => 21, + 'w' => 22, + 'x' => 23, + 'y' => 24, + 'z' => 25, + //Base10 digits + '2' => 26, + '3' => 27, + '4' => 28, + '5' => 29, + '6' => 30, + '7' => 31, + + _=> throw new FormatException("Character found is not a Base32 encoded character") + }; + } + + /// <summary> + /// Calculates the maximum buffer size required to encode a binary block to its Base32 + /// character encoding + /// </summary> + /// <param name="bufferSize">The binary buffer size used to calculate the base32 buffer size</param> + /// <returns>The maximum size (including padding) of the character buffer required to encode the binary data</returns> + public static int Base32CalcMaxBufferSize(int bufferSize) + { + /* + * Base32 encoding consumes 8 bytes for every 5 bytes + * of input data + */ + //Add up to 8 bytes for padding + return (int)(Math.Ceiling(bufferSize / 5d) * 8) + (8 - (bufferSize % 8)); + } + + /// <summary> + /// Converts the binary buffer to a base32 character string with optional padding characters + /// </summary> + /// <param name="binBuffer">The buffer to encode</param> + /// <param name="withPadding">Should padding be included in the result</param> + /// <returns>The base32 encoded string representation of the specified buffer</returns> + /// <exception cref="InternalBufferTooSmallException"></exception> + public static string ToBase32String(ReadOnlySpan<byte> binBuffer, bool withPadding = false) + { + string value; + //Calculate the base32 entropy to alloc an appropriate buffer (minium buffer of 2 chars) + int entropy = Base32CalcMaxBufferSize(binBuffer.Length); + //Alloc buffer for enough size (2*long bytes) is not an issue + using (UnsafeMemoryHandle<char> charBuffer = Memory.Memory.UnsafeAlloc<char>(entropy)) + { + //Encode + ERRNO encoded = TryToBase32Chars(binBuffer, charBuffer.Span); + if (!encoded) + { + throw new InternalBufferTooSmallException("Base32 char buffer was too small"); + } + //Convert with or w/o padding + if (withPadding) + { + value = charBuffer.Span[0..(int)encoded].ToString(); + } + else + { + value = charBuffer.Span[0..(int)encoded].Trim('=').ToString(); + } + } + return value; + } + /// <summary> + /// Converts the base32 character buffer to its structure representation + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="base32">The base32 character buffer</param> + /// <returns>The new structure of the base32 data</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="InternalBufferTooSmallException"></exception> + public static T FromBase32String<T>(ReadOnlySpan<char> base32) where T: unmanaged + { + //calc size of bin buffer + int size = base32.Length; + //Rent a bin buffer + using UnsafeMemoryHandle<byte> binBuffer = Memory.Memory.UnsafeAlloc<byte>(size); + //Try to decode the data + ERRNO decoded = TryFromBase32Chars(base32, binBuffer.Span); + //Marshal back to a struct + return decoded ? MemoryMarshal.Read<T>(binBuffer.Span[..(int)decoded]) : throw new InternalBufferTooSmallException("Binbuffer was too small"); + } + + /// <summary> + /// Gets a byte array of the base32 decoded data + /// </summary> + /// <param name="base32">The character array to decode</param> + /// <returns>The byte[] of the decoded binary data, or null if the supplied character array was empty</returns> + /// <exception cref="InternalBufferTooSmallException"></exception> + public static byte[]? FromBase32String(ReadOnlySpan<char> base32) + { + if (base32.IsEmpty) + { + return null; + } + //Buffer size of the base32 string will always be enough buffer space + using UnsafeMemoryHandle<byte> tempBuffer = Memory.Memory.UnsafeAlloc<byte>(base32.Length); + //Try to decode the data + ERRNO decoded = TryFromBase32Chars(base32, tempBuffer.Span); + + return decoded ? tempBuffer.Span[0..(int)decoded].ToArray() : throw new InternalBufferTooSmallException("Binbuffer was too small"); + } + + /// <summary> + /// Converts a structure to its base32 representation and returns the string of its value + /// </summary> + /// <typeparam name="T">The structure type</typeparam> + /// <param name="value">The structure to encode</param> + /// <param name="withPadding">A value indicating if padding should be used</param> + /// <returns>The base32 string representation of the structure</returns> + /// <exception cref="ArgumentException"></exception> + /// <exception cref="InternalBufferTooSmallException"></exception> + public static string ToBase32String<T>(T value, bool withPadding = false) where T : unmanaged + { + //get the size of the structure + int binSize = Unsafe.SizeOf<T>(); + //Rent a bin buffer + Span<byte> binBuffer = stackalloc byte[binSize]; + //Write memory to buffer + MemoryMarshal.Write(binBuffer, ref value); + //Convert to base32 + return ToBase32String(binBuffer, withPadding); + } + + #endregion + + #region percent encoding + + private static readonly ReadOnlyMemory<byte> HexToUtf8Pos = new byte[16] + { + 0x30, //0 + 0x31, //1 + 0x32, //2 + 0x33, //3 + 0x34, //4 + 0x35, //5 + 0x36, //6 + 0x37, //7 + 0x38, //8 + 0x39, //9 + + 0x41, //A + 0x42, //B + 0x43, //C + 0x44, //D + 0x45, //E + 0x46 //F + }; + + /// <summary> + /// Deterimes the size of the buffer needed to encode a utf8 encoded + /// character buffer into its url-safe percent/hex encoded representation + /// </summary> + /// <param name="utf8Bytes">The buffer to examine</param> + /// <param name="allowedChars">A sequence of characters that are excluded from encoding</param> + /// <returns>The size of the buffer required to encode</returns> + public static unsafe int PercentEncodeCalcBufferSize(ReadOnlySpan<byte> utf8Bytes, in ReadOnlySpan<byte> allowedChars = default) + { + /* + * For every illegal character, the percent encoding adds 3 bytes of + * entropy. So a single byte will be replaced by 3, so adding + * 2 bytes for every illegal character plus the length of the + * intial buffer, we get the size of the buffer needed to + * percent encode. + */ + int count = 0, len = utf8Bytes.Length; + fixed (byte* utfBase = &MemoryMarshal.GetReference(utf8Bytes)) + { + //Find all unsafe characters and add the entropy size + for (int i = 0; i < len; i++) + { + if (!IsUrlSafeChar(utfBase[i], in allowedChars)) + { + count += 2; + } + } + } + //Size is initial buffer size + count bytes + return len + count; + } + + /// <summary> + /// Percent encodes the buffer for utf8 encoded characters to its percent/hex encoded + /// utf8 character representation + /// </summary> + /// <param name="utf8Bytes">The buffer of utf8 encoded characters to encode</param> + /// <param name="utf8Output">The buffer to write the encoded characters to</param> + /// <param name="allowedChars">A sequence of characters that are excluded from encoding</param> + /// <returns>The number of characters encoded and written to the output buffer</returns> + public static ERRNO PercentEncode(ReadOnlySpan<byte> utf8Bytes, Span<byte> utf8Output, in ReadOnlySpan<byte> allowedChars = default) + { + int outPos = 0, len = utf8Bytes.Length; + ReadOnlySpan<byte> lookupTable = HexToUtf8Pos.Span; + for (int i = 0; i < len; i++) + { + byte value = utf8Bytes[i]; + //Check if value is url safe + if(IsUrlSafeChar(value, in allowedChars)) + { + //Skip + utf8Output[outPos++] = value; + } + else + { + //Percent encode + utf8Output[outPos++] = 0x25; // '%' + //Calc and store the encoded by the upper 4 bits + utf8Output[outPos++] = lookupTable[(value & 0xf0) >> 4]; + //Store lower 4 bits in encoded value + utf8Output[outPos++] = lookupTable[value & 0x0f]; + } + } + //Return the size of the output buffer + return outPos; + } + + private static bool IsUrlSafeChar(byte value, in ReadOnlySpan<byte> allowedChars) + { + return + // base10 digits + value > 0x2f && value < 0x3a + // '_' (underscore) + || value == 0x5f + // '-' (hyphen) + || value == 0x2d + // Uppercase letters + || value > 0x40 && value < 0x5b + // lowercase letters + || value > 0x60 && value < 0x7b + // Check allowed characters + || allowedChars.Contains(value); + } + + //TODO: Implement decode with better performance, lookup table or math vs searching the table + + /// <summary> + /// Decodes a percent (url/hex) encoded utf8 encoded character buffer to its utf8 + /// encoded binary value + /// </summary> + /// <param name="utf8Encoded">The buffer containg characters to be decoded</param> + /// <param name="utf8Output">The buffer to write deocded values to</param> + /// <returns>The nuber of bytes written to the output buffer</returns> + /// <exception cref="FormatException"></exception> + public static ERRNO PercentDecode(ReadOnlySpan<byte> utf8Encoded, Span<byte> utf8Output) + { + int outPos = 0, len = utf8Encoded.Length; + ReadOnlySpan<byte> lookupTable = HexToUtf8Pos.Span; + for (int i = 0; i < len; i++) + { + byte value = utf8Encoded[i]; + //Begining of percent encoding character + if(value == 0x25) + { + //Calculate the base16 multiplier from the upper half of the + int multiplier = lookupTable.IndexOf(utf8Encoded[i + 1]); + //get the base16 lower half to add + int lower = lookupTable.IndexOf(utf8Encoded[i + 2]); + //Check format + if(multiplier < 0 || lower < 0) + { + throw new FormatException($"Encoded buffer contains invalid hexadecimal characters following the % character at position {i}"); + } + //Calculate the new value, shift multiplier to the upper 4 bits, then mask + or the lower 4 bits + value = (byte)(((byte)(multiplier << 4)) | ((byte)lower & 0x0f)); + //Advance the encoded index by the two consumed chars + i += 2; + } + utf8Output[outPos++] = value; + } + return outPos; + } + + #endregion + + #region Base64 + + /// <summary> + /// Tries to convert the specified span containing a string representation that is + /// encoded with base-64 digits into a span of 8-bit unsigned integers. + /// </summary> + /// <param name="base64">Base64 character data to recover</param> + /// <param name="buffer">The binary output buffer to write converted characters to</param> + /// <returns>The number of bytes written, or <see cref="ERRNO.E_FAIL"/> of the conversion was unsucessful</returns> + public static ERRNO TryFromBase64Chars(ReadOnlySpan<char> base64, Span<byte> buffer) + { + return Convert.TryFromBase64Chars(base64, buffer, out int bytesWritten) ? bytesWritten : ERRNO.E_FAIL; + } + /// <summary> + /// Tries to convert the 8-bit unsigned integers inside the specified read-only span + /// into their equivalent string representation that is encoded with base-64 digits. + /// You can optionally specify whether to insert line breaks in the return value. + /// </summary> + /// <param name="buffer">The binary buffer to convert characters from</param> + /// <param name="base64">The base64 output buffer</param> + /// <param name="options"> + /// One of the enumeration values that specify whether to insert line breaks in the + /// return value. The default value is System.Base64FormattingOptions.None. + /// </param> + /// <returns>The number of characters encoded, or <see cref="ERRNO.E_FAIL"/> if conversion was unsuccessful</returns> + public static ERRNO TryToBase64Chars(ReadOnlySpan<byte> buffer, Span<char> base64, Base64FormattingOptions options = Base64FormattingOptions.None) + { + return Convert.TryToBase64Chars(buffer, base64, out int charsWritten, options: options) ? charsWritten : ERRNO.E_FAIL; + } + + + /* + * Calc base64 padding chars excluding the length mod 4 = 0 case + * by and-ing 0x03 (011) with the result + */ + + /// <summary> + /// Determines the number of missing padding bytes from the length of the base64 + /// data sequence. + /// <code> + /// Formula (4 - (length mod 4) and 0x03 + /// </code> + /// </summary> + /// <param name="length">The length of the base64 buffer</param> + /// <returns>The number of padding bytes to add to the end of the sequence</returns> + public static int Base64CalcRequiredPadding(int length) => (4 - (length % 4)) & 0x03; + + /// <summary> + /// Converts a base64 utf8 encoded binary buffer to + /// its base64url encoded version + /// </summary> + /// <param name="base64">The binary buffer to convert</param> + public static unsafe void Base64ToUrlSafeInPlace(Span<byte> base64) + { + int len = base64.Length; + + fixed(byte* ptr = &MemoryMarshal.GetReference(base64)) + { + for (int i = 0; i < len; i++) + { + switch (ptr[i]) + { + //Replace + with - (minus) + case 0x2b: + ptr[i] = 0x2d; + break; + //Replace / with _ (underscore) + case 0x2f: + ptr[i] = 0x5f; + break; + } + } + } + } + /// <summary> + /// Converts a base64url encoded utf8 encoded binary buffer to + /// its base64 encoded version + /// </summary> + /// <param name="uft8Base64Url">The base64url utf8 to decode</param> + public static unsafe void Base64FromUrlSafeInPlace(Span<byte> uft8Base64Url) + { + int len = uft8Base64Url.Length; + + fixed (byte* ptr = &MemoryMarshal.GetReference(uft8Base64Url)) + { + for (int i = 0; i < len; i++) + { + switch (ptr[i]) + { + //Replace - with + (plus) + case 0x2d: + ptr[i] = 0x2b; + break; + //Replace _ with / (slash) + case 0x5f: + ptr[i] = 0x2f; + break; + } + } + } + } + + /// <summary> + /// Converts the input buffer to a url safe base64 encoded + /// utf8 buffer from the base64 input buffer. The base64 is copied + /// directly to the output then converted in place. This is + /// just a shortcut method for readonly spans + /// </summary> + /// <param name="base64">The base64 encoded data</param> + /// <param name="base64Url">The base64url encoded output</param> + /// <returns>The size of the <paramref name="base64"/> buffer</returns> + public static ERRNO Base64ToUrlSafe(ReadOnlySpan<byte> base64, Span<byte> base64Url) + { + //Aligned copy to the output buffer + base64.CopyTo(base64Url); + //One time convert the output buffer to url safe + Base64ToUrlSafeInPlace(base64Url); + return base64.Length; + } + + /// <summary> + /// Converts the urlsafe input buffer to a base64 encoded + /// utf8 buffer from the base64 input buffer. The base64 is copied + /// directly to the output then converted in place. This is + /// just a shortcut method for readonly spans + /// </summary> + /// <param name="base64">The base64 encoded data</param> + /// <param name="base64Url">The base64url encoded output</param> + /// <returns>The size of the <paramref name="base64Url"/> buffer</returns> + public static ERRNO Base64FromUrlSafe(ReadOnlySpan<byte> base64Url, Span<byte> base64) + { + //Aligned copy to the output buffer + base64Url.CopyTo(base64); + //One time convert the output buffer to url safe + Base64FromUrlSafeInPlace(base64); + return base64Url.Length; + } + + /// <summary> + /// Decodes a utf8 base64url encoded sequence of data and writes it + /// to the supplied output buffer + /// </summary> + /// <param name="utf8Base64Url">The utf8 base64 url encoded string</param> + /// <param name="output">The output buffer to write the decoded data to</param> + /// <returns>The number of bytes written or <see cref="ERRNO.E_FAIL"/> if the operation failed</returns> + public static ERRNO Base64UrlDecode(ReadOnlySpan<byte> utf8Base64Url, Span<byte> output) + { + if(utf8Base64Url.IsEmpty || output.IsEmpty) + { + return ERRNO.E_FAIL; + } + //url deocde + ERRNO count = Base64FromUrlSafe(utf8Base64Url, output); + + //Writer for adding padding bytes + ForwardOnlyWriter<byte> writer = new (output); + writer.Advance(count); + + //Calc required padding + int paddingToAdd = Base64CalcRequiredPadding(writer.Written); + //Add padding bytes + for (; paddingToAdd > 0; paddingToAdd--) + { + writer.Append(0x3d); // '=' + } + + //Base64 decode in place, we should have a buffer large enough + OperationStatus status = Base64.DecodeFromUtf8InPlace(writer.AsSpan(), out int bytesWritten); + //If status is successful return the number of bytes written + return status == OperationStatus.Done ? bytesWritten : ERRNO.E_FAIL; + } + + /// <summary> + /// Decodes a base64url encoded character sequence + /// of data and writes it to the supplied output buffer + /// </summary> + /// <param name="chars">The character buffer to decode</param> + /// <param name="output">The output buffer to write decoded data to</param> + /// <param name="encoding">The character encoding</param> + /// <returns>The number of bytes written or <see cref="ERRNO.E_FAIL"/> if the operation failed</returns> + /// <exception cref="InternalBufferTooSmallException"></exception> + public static ERRNO Base64UrlDecode(ReadOnlySpan<char> chars, Span<byte> output, Encoding? encoding = null) + { + if (chars.IsEmpty || output.IsEmpty) + { + return ERRNO.E_FAIL; + } + //Set the encoding to utf8 + encoding ??= Encoding.UTF8; + //get the number of bytes to alloc a buffer + int decodedSize = encoding.GetByteCount(chars); + + //alloc buffer + using UnsafeMemoryHandle<byte> decodeHandle = Memory.Memory.UnsafeAlloc<byte>(decodedSize); + //Get the utf8 binary data + int count = encoding.GetBytes(chars, decodeHandle); + return Base64UrlDecode(decodeHandle.Span[..count], output); + } + + #endregion + } +}
\ No newline at end of file diff --git a/Utils/tests/ERRNOTest.cs b/Utils/tests/ERRNOTest.cs new file mode 100644 index 0000000..bd14b50 --- /dev/null +++ b/Utils/tests/ERRNOTest.cs @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.UtilsTests +* File: ERRNOTest.cs +* +* ERRNOTest.cs is part of VNLib.UtilsTests which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.UtilsTests is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.UtilsTests is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.UtilsTests. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading.Tasks; + +using Microsoft.VisualStudio.TestTools.UnitTesting; + +using VNLib.Utils; + +namespace VNLib.Utils.Tests +{ + [TestClass] + public class ERRNOTest + { + + [TestMethod] + public unsafe void ERRNOSizeTest() + { + Assert.IsTrue(sizeof(ERRNO) == sizeof(nint)); + } + + } +} diff --git a/Utils/tests/Memory/MemoryHandleTest.cs b/Utils/tests/Memory/MemoryHandleTest.cs new file mode 100644 index 0000000..02ef1f1 --- /dev/null +++ b/Utils/tests/Memory/MemoryHandleTest.cs @@ -0,0 +1,183 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.UtilsTests +* File: MemoryHandleTest.cs +* +* MemoryHandleTest.cs is part of VNLib.UtilsTests which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.UtilsTests is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.UtilsTests is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.UtilsTests. If not, see http://www.gnu.org/licenses/. +*/ + + +using Microsoft.VisualStudio.TestTools.UnitTesting; + +using VNLib.Utils; +using VNLib.Utils.Extensions; + +using static VNLib.Utils.Memory.Memory; + +namespace VNLib.Utils.Memory.Tests +{ + [TestClass] + public class MemoryHandleTest + { + + [TestMethod] + public void MemoryHandleAllocLongExtensionTest() + { + //Check for negatives + Assert.ThrowsException<ArgumentOutOfRangeException>(() => Shared.Alloc<byte>(-1)); + + //Make sure over-alloc throws + Assert.ThrowsException<NativeMemoryOutOfMemoryException>(() => Shared.Alloc<byte>(ulong.MaxValue, false)); + } +#if TARGET_64_BIT + [TestMethod] + public unsafe void MemoryHandleBigAllocTest() + { + const long bigHandleSize = (long)uint.MaxValue + 1024; + + using MemoryHandle<byte> handle = Shared.Alloc<byte>(bigHandleSize); + + //verify size + Assert.AreEqual(handle.ByteLength, (ulong)bigHandleSize); + //Since handle is byte, should also match + Assert.AreEqual(handle.Length, (ulong)bigHandleSize); + + //Should throw overflow + Assert.ThrowsException<OverflowException>(() => _ = handle.Span); + Assert.ThrowsException<OverflowException>(() => _ = handle.IntLength); + + //Should get the remaining span + Span<byte> offsetTest = handle.GetOffsetSpan(int.MaxValue, 1024); + + Assert.ThrowsException<ArgumentOutOfRangeException>(() => _ = handle.GetOffsetSpan((long)int.MaxValue + 1, 1024)); + + } +#else + +#endif + + [TestMethod] + public unsafe void BasicMemoryHandleTest() + { + using MemoryHandle<byte> handle = Shared.Alloc<byte>(128, true); + + Assert.AreEqual(handle.IntLength, 128); + + Assert.AreEqual(handle.Length, (ulong)128); + + //Check span against base pointer deref + + handle.Span[120] = 10; + + Assert.AreEqual(*handle.GetOffset(120), 10); + } + + + [TestMethod] + public unsafe void MemoryHandleDisposedTest() + { + using MemoryHandle<byte> handle = Shared.Alloc<byte>(1024); + + //Make sure handle is not invalid until disposed + Assert.IsFalse(handle.IsInvalid); + Assert.IsFalse(handle.IsClosed); + Assert.AreNotEqual(IntPtr.Zero, handle.BasePtr); + + //Dispose the handle early and test + handle.Dispose(); + + Assert.IsTrue(handle.IsInvalid); + Assert.IsTrue(handle.IsClosed); + + Assert.ThrowsException<ObjectDisposedException>(() => _ = handle.Span); + Assert.ThrowsException<ObjectDisposedException>(() => _ = handle.BasePtr); + Assert.ThrowsException<ObjectDisposedException>(() => _ = handle.Base); + Assert.ThrowsException<ObjectDisposedException>(() => handle.Resize(10)); + Assert.ThrowsException<ObjectDisposedException>(() => _ = handle.GetOffset(10)); + Assert.ThrowsException<ObjectDisposedException>(() => handle.ThrowIfClosed()); + } + + [TestMethod] + public unsafe void MemoryHandleCountDisposedTest() + { + using MemoryHandle<byte> handle = Shared.Alloc<byte>(1024); + + //Make sure handle is not invalid until disposed + Assert.IsFalse(handle.IsInvalid); + Assert.IsFalse(handle.IsClosed); + Assert.AreNotEqual(IntPtr.Zero, handle.BasePtr); + + bool test = false; + //Increase handle counter + handle.DangerousAddRef(ref test); + Assert.IsTrue(test); + + //Dispose the handle early and test + handle.Dispose(); + + //Asser is valid still + + //Make sure handle is not invalid until disposed + Assert.IsFalse(handle.IsInvalid); + Assert.IsFalse(handle.IsClosed); + Assert.AreNotEqual(IntPtr.Zero, handle.BasePtr); + + //Dec handle count + handle.DangerousRelease(); + + //Now make sure the class is disposed + + Assert.IsTrue(handle.IsInvalid); + Assert.IsTrue(handle.IsClosed); + Assert.ThrowsException<ObjectDisposedException>(() => _ = handle.Span); + } + + [TestMethod] + public unsafe void MemoryHandleExtensionsTest() + { + using MemoryHandle<byte> handle = Shared.Alloc<byte>(1024); + + Assert.AreEqual(handle.IntLength, 1024); + + Assert.ThrowsException<ArgumentOutOfRangeException>(() => handle.Resize(-1)); + + //Resize the handle + handle.Resize(2048); + + Assert.AreEqual(handle.IntLength, 2048); + + Assert.IsTrue(handle.AsSpan(2048).IsEmpty); + + Assert.ThrowsException<ArgumentOutOfRangeException>(() => _ = handle.AsSpan(2049)); + + Assert.ThrowsException<ArgumentOutOfRangeException>(() => _ = handle.GetOffset(2049)); + + Assert.ThrowsException<ArgumentOutOfRangeException>(() => _ = handle.GetOffset(-1)); + + //test resize + handle.ResizeIfSmaller(100); + //Handle should be unmodified + Assert.AreEqual(handle.IntLength, 2048); + + //test working + handle.ResizeIfSmaller(4096); + Assert.AreEqual(handle.IntLength, 4096); + } + } +} diff --git a/Utils/tests/Memory/MemoryTests.cs b/Utils/tests/Memory/MemoryTests.cs new file mode 100644 index 0000000..5b68cf5 --- /dev/null +++ b/Utils/tests/Memory/MemoryTests.cs @@ -0,0 +1,244 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.UtilsTests +* File: MemoryTests.cs +* +* MemoryTests.cs is part of VNLib.UtilsTests which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.UtilsTests is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.UtilsTests is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.UtilsTests. If not, see http://www.gnu.org/licenses/. +*/ + +using Microsoft.VisualStudio.TestTools.UnitTesting; +using System.Runtime.InteropServices; + +using VNLib.Utils.Extensions; + +namespace VNLib.Utils.Memory.Tests +{ + [TestClass()] + public class MemoryTests + { + [TestMethod] + public void MemorySharedHeapLoadedTest() + { + Assert.IsNotNull(Memory.Shared); + } + + [TestMethod()] + public void UnsafeAllocTest() + { + //test against negative number + Assert.ThrowsException<ArgumentException>(() => Memory.UnsafeAlloc<byte>(-1)); + + //Alloc large block test (100mb) + const int largTestSize = 100000 * 1024; + //Alloc super small block + const int smallTestSize = 5; + + using (UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(largTestSize, false)) + { + Assert.AreEqual(largTestSize, buffer.IntLength); + Assert.AreEqual(largTestSize, buffer.Span.Length); + + buffer.Span[0] = 254; + Assert.AreEqual(buffer.Span[0], 254); + } + + using (UnsafeMemoryHandle<byte> buffer = Memory.UnsafeAlloc<byte>(smallTestSize, false)) + { + Assert.AreEqual(smallTestSize, buffer.IntLength); + Assert.AreEqual(smallTestSize, buffer.Span.Length); + + buffer.Span[0] = 254; + Assert.AreEqual(buffer.Span[0], 254); + } + + //Different data type + + using(UnsafeMemoryHandle<long> buffer = Memory.UnsafeAlloc<long>(largTestSize, false)) + { + Assert.AreEqual(largTestSize, buffer.IntLength); + Assert.AreEqual(largTestSize, buffer.Span.Length); + + buffer.Span[0] = long.MaxValue; + Assert.AreEqual(buffer.Span[0], long.MaxValue); + } + + using (UnsafeMemoryHandle<long> buffer = Memory.UnsafeAlloc<long>(smallTestSize, false)) + { + Assert.AreEqual(smallTestSize, buffer.IntLength); + Assert.AreEqual(smallTestSize, buffer.Span.Length); + + buffer.Span[0] = long.MaxValue; + Assert.AreEqual(buffer.Span[0], long.MaxValue); + } + } + + [TestMethod()] + public void UnsafeZeroMemoryAsSpanTest() + { + //Alloc test buffer + Span<byte> test = new byte[1024]; + test.Fill(0); + //test other empty span + Span<byte> verify = new byte[1024]; + verify.Fill(0); + + //Fill test buffer with random values + Random.Shared.NextBytes(test); + + //make sure buffers are not equal + Assert.IsFalse(test.SequenceEqual(verify)); + + //Zero buffer + Memory.UnsafeZeroMemory<byte>(test); + + //Make sure buffers are equal + Assert.IsTrue(test.SequenceEqual(verify)); + } + + [TestMethod()] + public void UnsafeZeroMemoryAsMemoryTest() + { + //Alloc test buffer + Memory<byte> test = new byte[1024]; + test.Span.Fill(0); + //test other empty span + Memory<byte> verify = new byte[1024]; + verify.Span.Fill(0); + + //Fill test buffer with random values + Random.Shared.NextBytes(test.Span); + + //make sure buffers are not equal + Assert.IsFalse(test.Span.SequenceEqual(verify.Span)); + + //Zero buffer + Memory.UnsafeZeroMemory<byte>(test); + + //Make sure buffers are equal + Assert.IsTrue(test.Span.SequenceEqual(verify.Span)); + } + + [TestMethod()] + public void InitializeBlockAsSpanTest() + { + //Alloc test buffer + Span<byte> test = new byte[1024]; + test.Fill(0); + //test other empty span + Span<byte> verify = new byte[1024]; + verify.Fill(0); + + //Fill test buffer with random values + Random.Shared.NextBytes(test); + + //make sure buffers are not equal + Assert.IsFalse(test.SequenceEqual(verify)); + + //Zero buffer + Memory.InitializeBlock(test); + + //Make sure buffers are equal + Assert.IsTrue(test.SequenceEqual(verify)); + } + + [TestMethod()] + public void InitializeBlockMemoryTest() + { + //Alloc test buffer + Memory<byte> test = new byte[1024]; + test.Span.Fill(0); + //test other empty span + Memory<byte> verify = new byte[1024]; + verify.Span.Fill(0); + + //Fill test buffer with random values + Random.Shared.NextBytes(test.Span); + + //make sure buffers are not equal + Assert.IsFalse(test.Span.SequenceEqual(verify.Span)); + + //Zero buffer + Memory.InitializeBlock(test); + + //Make sure buffers are equal + Assert.IsTrue(test.Span.SequenceEqual(verify.Span)); + } + + #region structmemory tests + + [StructLayout(LayoutKind.Sequential)] + struct TestStruct + { + public int X; + public int Y; + } + + [TestMethod()] + public unsafe void ZeroStructAsPointerTest() + { + TestStruct* s = Memory.Shared.StructAlloc<TestStruct>(); + s->X = 10; + s->Y = 20; + Assert.AreEqual(10, s->X); + Assert.AreEqual(20, s->Y); + //zero struct + Memory.ZeroStruct(s); + //Verify data was zeroed + Assert.AreEqual(0, s->X); + Assert.AreEqual(0, s->Y); + //Free struct + Memory.Shared.StructFree(s); + } + + [TestMethod()] + public unsafe void ZeroStructAsVoidPointerTest() + { + TestStruct* s = Memory.Shared.StructAlloc<TestStruct>(); + s->X = 10; + s->Y = 20; + Assert.AreEqual(10, s->X); + Assert.AreEqual(20, s->Y); + //zero struct + Memory.ZeroStruct<TestStruct>((void*)s); + //Verify data was zeroed + Assert.AreEqual(0, s->X); + Assert.AreEqual(0, s->Y); + //Free struct + Memory.Shared.StructFree(s); + } + + [TestMethod()] + public unsafe void ZeroStructAsIntPtrTest() + { + TestStruct* s = Memory.Shared.StructAlloc<TestStruct>(); + s->X = 10; + s->Y = 20; + Assert.AreEqual(10, s->X); + Assert.AreEqual(20, s->Y); + //zero struct + Memory.ZeroStruct<TestStruct>((IntPtr)s); + //Verify data was zeroed + Assert.AreEqual(0, s->X); + Assert.AreEqual(0, s->Y); + //Free struct + Memory.Shared.StructFree(s); + } + #endregion + } +}
\ No newline at end of file diff --git a/Utils/tests/Memory/VnTableTests.cs b/Utils/tests/Memory/VnTableTests.cs new file mode 100644 index 0000000..11350d4 --- /dev/null +++ b/Utils/tests/Memory/VnTableTests.cs @@ -0,0 +1,168 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.UtilsTests +* File: VnTableTests.cs +* +* VnTableTests.cs is part of VNLib.UtilsTests which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.UtilsTests is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.UtilsTests is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.UtilsTests. If not, see http://www.gnu.org/licenses/. +*/ + +using Microsoft.VisualStudio.TestTools.UnitTesting; + + +namespace VNLib.Utils.Memory.Tests +{ + [TestClass()] + public class VnTableTests + { + [TestMethod()] + public void VnTableTest() + { + Assert.ThrowsException<ArgumentOutOfRangeException>(() => + { + using VnTable<int> table = new(-1, 0); + }); + Assert.ThrowsException<ArgumentOutOfRangeException>(() => + { + using VnTable<int> table = new(0, -1); + }); + Assert.ThrowsException<ArgumentOutOfRangeException>(() => + { + using VnTable<int> table = new(-1, -1); + }); + + //Empty table + using (VnTable<int> empty = new(0, 0)) + { + Assert.IsTrue(empty.Empty); + //Test 0 rows/cols + Assert.AreEqual(0, empty.Rows); + Assert.AreEqual(0, empty.Cols); + } + + using (VnTable<int> table = new(40000, 10000)) + { + Assert.IsFalse(table.Empty); + + //Test table size + Assert.AreEqual(40000, table.Rows); + Assert.AreEqual(10000, table.Cols); + } + + + //Test oom, should be native + Assert.ThrowsException<NativeMemoryOutOfMemoryException>(() => + { + using VnTable<int> table = new(int.MaxValue, 2); + }); + } + + [TestMethod()] + public void VnTableTest1() + { + //No throw if empty + using VnTable<int> table = new(null!,0, 0); + + //Throw if table is not empty + Assert.ThrowsException<ArgumentNullException>(() => + { + using VnTable<int> table = new(null!,1, 1); + }); + + } + + [TestMethod()] + public void GetSetTest() + { + static void TestIndexAt(VnTable<int> table, int row, int col, int value) + { + table[row, col] = value; + Assert.AreEqual(value, table[row, col]); + Assert.AreEqual(value, table.Get(row, col)); + } + + static void TestSetAt(VnTable<int> table, int row, int col, int value) + { + table.Set(row, col, value); + Assert.AreEqual(value, table[row, col]); + Assert.AreEqual(value, table.Get(row, col)); + } + + static void TestSetDirectAccess(VnTable<int> table, int row, int col, int value) + { + int address = row * table.Cols + col; + table[(uint)address] = value; + + //Get value using indexer + Assert.AreEqual(value, table[row, col]); + } + + static void TestGetDirectAccess(VnTable<int> table, int row, int col, int value) + { + table[row, col] = value; + + int address = row * table.Cols + col; + + //Test direct access + Assert.AreEqual(value, table[(uint)address]); + + //Get value using indexer + Assert.AreEqual(value, table[row, col]); + Assert.AreEqual(value, table.Get(row, col)); + } + + + using (VnTable<int> table = new(11, 11)) + { + //Test index at 10,10 + TestIndexAt(table, 10, 10, 11); + //Test same index with different value using the .set() method + TestSetAt(table, 10, 10, 25); + + //Test direct access + TestSetDirectAccess(table, 10, 10, 50); + + TestGetDirectAccess(table, 10, 10, 37); + + //Test index at 0,0 + TestIndexAt(table, 0, 0, 13); + TestSetAt(table, 0, 0, 85); + + //Test at 0,0 + TestSetDirectAccess(table, 0, 0, 100); + TestGetDirectAccess(table, 0, 0, 86); + } + } + + [TestMethod()] + public void DisposeTest() + { + //Alloc table + VnTable<int> table = new(10, 10); + //Dispose table + table.Dispose(); + + //Test that methods throw on access + Assert.ThrowsException<ObjectDisposedException>(() => table[10, 10] = 10); + Assert.ThrowsException<ObjectDisposedException>(() => table.Set(10, 10, 10)); + Assert.ThrowsException<ObjectDisposedException>(() => table[10, 10] == 10); + Assert.ThrowsException<ObjectDisposedException>(() => table.Get(10, 10)); + } + + } +}
\ No newline at end of file diff --git a/Utils/tests/README.md b/Utils/tests/README.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/Utils/tests/README.md diff --git a/Utils/tests/VNLib.UtilsTests.csproj b/Utils/tests/VNLib.UtilsTests.csproj new file mode 100644 index 0000000..89b3124 --- /dev/null +++ b/Utils/tests/VNLib.UtilsTests.csproj @@ -0,0 +1,36 @@ +<Project Sdk="Microsoft.NET.Sdk"> + + <PropertyGroup> + <TargetFramework>net6.0</TargetFramework> + <ImplicitUsings>enable</ImplicitUsings> + <Nullable>enable</Nullable> + + <IsPackable>false</IsPackable> + + <AllowUnsafeBlocks>true</AllowUnsafeBlocks> + + </PropertyGroup> + + <ItemGroup> + <PackageReference Include="ErrorProne.NET.CoreAnalyzers" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="ErrorProne.NET.Structs" Version="0.1.2"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.4.0" /> + <PackageReference Include="MSTest.TestAdapter" Version="2.2.10" /> + <PackageReference Include="MSTest.TestFramework" Version="2.2.10" /> + <PackageReference Include="coverlet.collector" Version="3.2.0"> + <PrivateAssets>all</PrivateAssets> + <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> + </PackageReference> + </ItemGroup> + + <ItemGroup> + <ProjectReference Include="..\src\VNLib.Utils.csproj" /> + </ItemGroup> + +</Project> diff --git a/Utils/tests/VnEncodingTests.cs b/Utils/tests/VnEncodingTests.cs new file mode 100644 index 0000000..a4e52f0 --- /dev/null +++ b/Utils/tests/VnEncodingTests.cs @@ -0,0 +1,100 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: VNLib.UtilsTests +* File: VnEncodingTests.cs +* +* VnEncodingTests.cs is part of VNLib.UtilsTests which is part of the larger +* VNLib collection of libraries and utilities. +* +* VNLib.UtilsTests is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* VNLib.UtilsTests is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with VNLib.UtilsTests. If not, see http://www.gnu.org/licenses/. +*/ + +using System; +using System.Buffers; +using System.Buffers.Text; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Threading.Tasks; + +using Microsoft.VisualStudio.TestTools.UnitTesting; +using VNLib.Utils; + +namespace VNLib.Utils.Tests +{ + [TestClass()] + public class VnEncodingTests + { + + [TestMethod()] + public void Base64ToUrlSafeInPlaceTest() + { + //Get randomd data to encode + byte[] dataToEncode = RandomNumberGenerator.GetBytes(64); + //Calc buffer size + int base64Output = Base64.GetMaxEncodedToUtf8Length(64); + + byte[] encodeBuffer = new byte[base64Output]; + //Base64 encode + OperationStatus status = Base64.EncodeToUtf8(dataToEncode, encodeBuffer, out _, out int bytesEncoded, true); + + Assert.IsTrue(status == OperationStatus.Done); + + Span<byte> encodeSpan = encodeBuffer.AsSpan(0, bytesEncoded); + + //Make sure some illegal characters are encoded + Assert.IsTrue(encodeSpan.Contains((byte)'+') || encodeSpan.Contains((byte)'/')); + + //Convert to url safe + VnEncoding.Base64ToUrlSafeInPlace(encodeSpan); + + //Make sure the illegal characters are gone + Assert.IsFalse(encodeSpan.Contains((byte)'+') || encodeSpan.Contains((byte)'/')); + } + + [TestMethod()] + public void Base64FromUrlSafeInPlaceTest() + { + //url safe base64 with known encoded characters + const string base64UrlSafe = "lZUABUd8q2BS7p8giysuC7PpEabAFBnMqBPL-9A-qgfR1lbTHQ4tMm8E8nimm2YAd5NGDIQ0vxfU9i5l53tF_WXa_H4vkHfzlv0Df-lLADJV7z8sn-8sfUGdaAiIS8_4OmVGnnY4-TppLMsVR6ov2t07HdOHPPsFFhSpBMXa2pwRveRATcxBA2XxVe09FOWgahhssNS7lU9eC7fRw7icD4ZoJcLSRBbxrjRmeVXKhPIaXR-4mnQ5-vqYzAr9S99CthgbAtVn_WjmDcda6pUB9JW9lp7ylDa9e1r_z39cihTXMOGaUSjVURJaWrNF8CkfW56_x2ODCBmZPov1YyEhww=="; + + //Convert to utf8 binary + byte[] utf8 = Encoding.UTF8.GetBytes(base64UrlSafe); + + //url decode + VnEncoding.Base64FromUrlSafeInPlace(utf8); + + //Confirm illegal chars have been converted back to base64 + Assert.IsFalse(utf8.Contains((byte)'_') || utf8.Contains((byte)'-')); + + //Decode in place to confrim its valid + OperationStatus status = Base64.DecodeFromUtf8InPlace(utf8, out int bytesWritten); + + Assert.IsFalse(status == OperationStatus.NeedMoreData); + Assert.IsFalse(status == OperationStatus.DestinationTooSmall); + Assert.IsFalse(status == OperationStatus.InvalidData); + } + + [TestMethod()] + public void TryToBase64CharsTest() + { + + } + + + } +}
\ No newline at end of file diff --git a/WinRpMalloc/LICENSE.txt b/WinRpMalloc/LICENSE.txt new file mode 100644 index 0000000..2848520 --- /dev/null +++ b/WinRpMalloc/LICENSE.txt @@ -0,0 +1,346 @@ +The software in this repository is licensed under the GNU GPL version 2.0 (or any later version). + +SPDX-License-Identifier: GPL-2.0-or-later + +License-Text: + +GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + VNLib.Hashing.Portable is a compact .NET managed cryptographic operation + utilities library. + Copyright (C) 2022 Vaughn Nugent + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License.
\ No newline at end of file diff --git a/WinRpMalloc/README.md b/WinRpMalloc/README.md new file mode 100644 index 0000000..4f72b66 --- /dev/null +++ b/WinRpMalloc/README.md @@ -0,0 +1 @@ +# WinRpMalloc
\ No newline at end of file diff --git a/WinRpMalloc/src/WinRpMalloc.vcxproj b/WinRpMalloc/src/WinRpMalloc.vcxproj new file mode 100644 index 0000000..b39deed --- /dev/null +++ b/WinRpMalloc/src/WinRpMalloc.vcxproj @@ -0,0 +1,188 @@ +<?xml version="1.0" encoding="utf-8"?> +<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> + <ItemGroup Label="ProjectConfigurations"> + <ProjectConfiguration Include="Debug|Win32"> + <Configuration>Debug</Configuration> + <Platform>Win32</Platform> + </ProjectConfiguration> + <ProjectConfiguration Include="Release|Win32"> + <Configuration>Release</Configuration> + <Platform>Win32</Platform> + </ProjectConfiguration> + <ProjectConfiguration Include="Debug|x64"> + <Configuration>Debug</Configuration> + <Platform>x64</Platform> + </ProjectConfiguration> + <ProjectConfiguration Include="Release|x64"> + <Configuration>Release</Configuration> + <Platform>x64</Platform> + </ProjectConfiguration> + </ItemGroup> + <PropertyGroup Label="Globals"> + <VCProjectVersion>16.0</VCProjectVersion> + <Keyword>Win32Proj</Keyword> + <ProjectGuid>{f5bfb8aa-a436-4a8d-94bc-9eff3ad8aa1d}</ProjectGuid> + <RootNamespace>WinRpMalloc</RootNamespace> + <WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion> + </PropertyGroup> + <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" /> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration"> + <ConfigurationType>DynamicLibrary</ConfigurationType> + <UseDebugLibraries>true</UseDebugLibraries> + <PlatformToolset>v143</PlatformToolset> + <CharacterSet>Unicode</CharacterSet> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration"> + <ConfigurationType>DynamicLibrary</ConfigurationType> + <UseDebugLibraries>false</UseDebugLibraries> + <PlatformToolset>v143</PlatformToolset> + <WholeProgramOptimization>true</WholeProgramOptimization> + <CharacterSet>Unicode</CharacterSet> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration"> + <ConfigurationType>DynamicLibrary</ConfigurationType> + <PlatformToolset>v143</PlatformToolset> + <UseOfMfc>false</UseOfMfc> + <UseDebugLibraries>true</UseDebugLibraries> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration"> + <ConfigurationType>DynamicLibrary</ConfigurationType> + <UseDebugLibraries>false</UseDebugLibraries> + <PlatformToolset>v143</PlatformToolset> + <WholeProgramOptimization>true</WholeProgramOptimization> + <CharacterSet>Unicode</CharacterSet> + </PropertyGroup> + <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" /> + <ImportGroup Label="ExtensionSettings"> + </ImportGroup> + <ImportGroup Label="Shared"> + </ImportGroup> + <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'"> + <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" /> + </ImportGroup> + <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'"> + <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" /> + </ImportGroup> + <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'"> + <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" /> + </ImportGroup> + <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'"> + <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" /> + </ImportGroup> + <PropertyGroup Label="UserMacros" /> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'"> + <LinkIncremental>true</LinkIncremental> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'"> + <LinkIncremental>false</LinkIncremental> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'"> + <CopyCppRuntimeToOutputDir>true</CopyCppRuntimeToOutputDir> + <OutDir>$(ProjectDir)$(Platform)\$(Configuration)\</OutDir> + <TargetName>rpmalloc</TargetName> + </PropertyGroup> + <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'"> + <LinkIncremental>false</LinkIncremental> + <TargetName>rpmalloc</TargetName> + </PropertyGroup> + <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'"> + <ClCompile> + <WarningLevel>Level3</WarningLevel> + <SDLCheck>true</SDLCheck> + <PreprocessorDefinitions>WIN32;_DEBUG;WINRPMALLOC_EXPORTS;_WINDOWS;_USRDLL;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <ConformanceMode>true</ConformanceMode> + <PrecompiledHeader>Use</PrecompiledHeader> + <PrecompiledHeaderFile>pch.h</PrecompiledHeaderFile> + </ClCompile> + <Link> + <SubSystem>Windows</SubSystem> + <GenerateDebugInformation>true</GenerateDebugInformation> + <EnableUAC>false</EnableUAC> + </Link> + </ItemDefinitionGroup> + <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'"> + <ClCompile> + <WarningLevel>Level3</WarningLevel> + <FunctionLevelLinking>true</FunctionLevelLinking> + <IntrinsicFunctions>true</IntrinsicFunctions> + <SDLCheck>true</SDLCheck> + <PreprocessorDefinitions>WIN32;NDEBUG;WINRPMALLOC_EXPORTS;_WINDOWS;_USRDLL;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <ConformanceMode>true</ConformanceMode> + <PrecompiledHeader>Use</PrecompiledHeader> + <PrecompiledHeaderFile>pch.h</PrecompiledHeaderFile> + </ClCompile> + <Link> + <SubSystem>Windows</SubSystem> + <EnableCOMDATFolding>true</EnableCOMDATFolding> + <OptimizeReferences>true</OptimizeReferences> + <GenerateDebugInformation>true</GenerateDebugInformation> + <EnableUAC>false</EnableUAC> + </Link> + </ItemDefinitionGroup> + <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'"> + <ClCompile> + <WarningLevel>EnableAllWarnings</WarningLevel> + <SDLCheck>true</SDLCheck> + <PreprocessorDefinitions>_DEBUG;_WINDOWS;_USRDLL;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <ConformanceMode>true</ConformanceMode> + <PrecompiledHeader>Use</PrecompiledHeader> + <PrecompiledHeaderFile>pch.h</PrecompiledHeaderFile> + <LanguageStandard_C>stdc17</LanguageStandard_C> + <CallingConvention>StdCall</CallingConvention> + <MultiProcessorCompilation>true</MultiProcessorCompilation> + <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed> + <GenerateXMLDocumentationFiles>false</GenerateXMLDocumentationFiles> + <GenerateSourceDependencies>true</GenerateSourceDependencies> + <BasicRuntimeChecks>Default</BasicRuntimeChecks> + <MinimalRebuild>false</MinimalRebuild> + <CompileAs>CompileAsC</CompileAs> + <RuntimeLibrary>MultiThreadedDebugDLL</RuntimeLibrary> + </ClCompile> + <Link> + <GenerateDebugInformation>true</GenerateDebugInformation> + <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> + <Version> + </Version> + <LargeAddressAware>true</LargeAddressAware> + <SubSystem>Windows</SubSystem> + </Link> + </ItemDefinitionGroup> + <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'"> + <ClCompile> + <WarningLevel>Level3</WarningLevel> + <FunctionLevelLinking>true</FunctionLevelLinking> + <IntrinsicFunctions>true</IntrinsicFunctions> + <SDLCheck>true</SDLCheck> + <PreprocessorDefinitions>NDEBUG;WINRPMALLOC_EXPORTS;_WINDOWS;_USRDLL;%(PreprocessorDefinitions)</PreprocessorDefinitions> + <ConformanceMode>true</ConformanceMode> + <PrecompiledHeader>Use</PrecompiledHeader> + <PrecompiledHeaderFile>pch.h</PrecompiledHeaderFile> + </ClCompile> + <Link> + <SubSystem>Windows</SubSystem> + <EnableCOMDATFolding>true</EnableCOMDATFolding> + <OptimizeReferences>true</OptimizeReferences> + <GenerateDebugInformation>true</GenerateDebugInformation> + <EnableUAC>false</EnableUAC> + </Link> + </ItemDefinitionGroup> + <ItemGroup> + <ClInclude Include="framework.h" /> + <ClInclude Include="pch.h" /> + <ClInclude Include="rpmalloc.h" /> + <ClInclude Include="rpnew.h" /> + </ItemGroup> + <ItemGroup> + <ClCompile Include="dllmain.c" /> + <ClCompile Include="pch.c"> + <PrecompiledHeader Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">Create</PrecompiledHeader> + <PrecompiledHeader Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">Create</PrecompiledHeader> + <PrecompiledHeader Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">Create</PrecompiledHeader> + <PrecompiledHeader Condition="'$(Configuration)|$(Platform)'=='Release|x64'">Create</PrecompiledHeader> + </ClCompile> + <ClCompile Include="rpmalloc.c" /> + </ItemGroup> + <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> + <ImportGroup Label="ExtensionTargets"> + </ImportGroup> +</Project>
\ No newline at end of file diff --git a/WinRpMalloc/src/dllmain.c b/WinRpMalloc/src/dllmain.c new file mode 100644 index 0000000..10ea3f5 --- /dev/null +++ b/WinRpMalloc/src/dllmain.c @@ -0,0 +1,27 @@ +// dllmain.cpp : Defines the entry point for the DLL application. + +#include "pch.h" + +BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) +{ + /* + * Taken from the malloc.c file for initializing the library. + * and thread events + */ + switch (ul_reason_for_call) + { + case DLL_PROCESS_ATTACH: + rpmalloc_initialize(); + break; + case DLL_THREAD_ATTACH: + rpmalloc_thread_initialize(); + break; + case DLL_THREAD_DETACH: + rpmalloc_thread_finalize(1); + break; + case DLL_PROCESS_DETACH: + rpmalloc_finalize(); + break; + } + return TRUE; +}
\ No newline at end of file diff --git a/WinRpMalloc/src/framework.h b/WinRpMalloc/src/framework.h new file mode 100644 index 0000000..573886e --- /dev/null +++ b/WinRpMalloc/src/framework.h @@ -0,0 +1,29 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: WinRpMalloc +* File: framework.h +* +* framework.h is part of WinRpMalloc which is part of the larger +* VNLib collection of libraries and utilities. +* +* WinRpMalloc is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* WinRpMalloc is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with WinRpMalloc. If not, see http://www.gnu.org/licenses/. +*/ + +#pragma once + +#define WIN32_LEAN_AND_MEAN // Exclude rarely-used stuff from Windows headers +// Windows Header Files +#include <windows.h> diff --git a/WinRpMalloc/src/pch.c b/WinRpMalloc/src/pch.c new file mode 100644 index 0000000..64b7eef --- /dev/null +++ b/WinRpMalloc/src/pch.c @@ -0,0 +1,5 @@ +// pch.cpp: source file corresponding to the pre-compiled header + +#include "pch.h" + +// When you are using pre-compiled headers, this source file is necessary for compilation to succeed. diff --git a/WinRpMalloc/src/pch.h b/WinRpMalloc/src/pch.h new file mode 100644 index 0000000..d8c2409 --- /dev/null +++ b/WinRpMalloc/src/pch.h @@ -0,0 +1,55 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: WinRpMalloc +* File: pch.h +* +* pch.h is part of WinRpMalloc which is part of the larger +* VNLib collection of libraries and utilities. +* +* WinRpMalloc is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* WinRpMalloc is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with WinRpMalloc. If not, see http://www.gnu.org/licenses/. +*/ + +// pch.h: This is a precompiled header file. +// Files listed below are compiled only once, improving build performance for future builds. +// This also affects IntelliSense performance, including code completion and many code browsing features. +// However, files listed here are ALL re-compiled if any one of them is updated between builds. +// Do not add files here that you will be updating frequently as this negates the performance advantage. + +#ifndef PCH_H +#define PCH_H + +#include "framework.h" +// add headers that you want to pre-compile here + +//Using firstclass heaps, define +#define RPMALLOC_FIRST_CLASS_HEAPS 1 + +/* +* Enabling adaptive thread cache because I am not using thread initilaizations +*/ +#define ENABLE_ADAPTIVE_THREAD_CACHE 1 + +#ifdef DEBUG + +ENABLE_VALIDATE_ARGS 1 +ENABLE_ASSERTS 1 +ENABLE_STATISTICS 1 + +#endif // DEBUG + +#include "rpmalloc.h" + +#endif //PCH_H diff --git a/WinRpMalloc/src/rpmalloc.c b/WinRpMalloc/src/rpmalloc.c new file mode 100644 index 0000000..9228cd2 --- /dev/null +++ b/WinRpMalloc/src/rpmalloc.c @@ -0,0 +1,3571 @@ +/* rpmalloc.c - Memory allocator - Public Domain - 2016-2020 Mattias Jansson + * + * This library provides a cross-platform lock free thread caching malloc implementation in C11. + * The latest source code is always available at + * + * https://github.com/mjansson/rpmalloc + * + * This library is put in the public domain; you can redistribute it and/or modify it without any restrictions. + * + */ + + +#include "pch.h" +#include "rpmalloc.h" + + +//////////// +/// +/// Build time configurable limits +/// +////// + +#if defined(__clang__) +#pragma clang diagnostic ignored "-Wunused-macros" +#pragma clang diagnostic ignored "-Wunused-function" +#if __has_warning("-Wreserved-identifier") +#pragma clang diagnostic ignored "-Wreserved-identifier" +#endif +#if __has_warning("-Wstatic-in-inline") +#pragma clang diagnostic ignored "-Wstatic-in-inline" +#endif +#elif defined(__GNUC__) +#pragma GCC diagnostic ignored "-Wunused-macros" +#pragma GCC diagnostic ignored "-Wunused-function" +#endif + +#if defined(__GNUC__) || defined(__clang__) +#if !defined(__has_builtin) +#define __has_builtin(b) 0 +#endif + +#if __has_builtin(__builtin_memcpy_inline) +#define _rpmalloc_memcpy_const(x, y, s) __builtin_memcpy_inline(x, y, s) +#else +#define _rpmalloc_memcpy_const(x, y, s) \ + do { \ + _Static_assert(__builtin_choose_expr(__builtin_constant_p(s), 1, 0), "len must be a constant integer"); \ + memcpy(x, y, s); \ + } while (0) +#endif + +#if __has_builtin(__builtin_memset_inline) +#define _rpmalloc_memset_const(x, y, s) __builtin_memset_inline(x, y, s) +#else +#define _rpmalloc_memset_const(x, y, s) \ + do { \ + _Static_assert(__builtin_choose_expr(__builtin_constant_p(s), 1, 0), "len must be a constant integer"); \ + memset(x, y, s); \ + } while (0) +#endif +#else +#define _rpmalloc_memcpy_const(x, y, s) memcpy(x, y, s) +#define _rpmalloc_memset_const(x, y, s) memset(x, y, s) +#endif + +#ifndef HEAP_ARRAY_SIZE +//! Size of heap hashmap +#define HEAP_ARRAY_SIZE 47 +#endif +#ifndef ENABLE_THREAD_CACHE +//! Enable per-thread cache +#define ENABLE_THREAD_CACHE 1 +#endif +#ifndef ENABLE_GLOBAL_CACHE +//! Enable global cache shared between all threads, requires thread cache +#define ENABLE_GLOBAL_CACHE 1 +#endif +#ifndef ENABLE_VALIDATE_ARGS +//! Enable validation of args to public entry points +#define ENABLE_VALIDATE_ARGS 0 +#endif +#ifndef ENABLE_STATISTICS +//! Enable statistics collection +#define ENABLE_STATISTICS 0 +#endif +#ifndef ENABLE_ASSERTS +//! Enable asserts +#define ENABLE_ASSERTS 0 +#endif +#ifndef ENABLE_OVERRIDE +//! Override standard library malloc/free and new/delete entry points +#define ENABLE_OVERRIDE 0 +#endif +#ifndef ENABLE_PRELOAD +//! Support preloading +#define ENABLE_PRELOAD 0 +#endif +#ifndef DISABLE_UNMAP +//! Disable unmapping memory pages (also enables unlimited cache) +#define DISABLE_UNMAP 0 +#endif +#ifndef ENABLE_UNLIMITED_CACHE +//! Enable unlimited global cache (no unmapping until finalization) +#define ENABLE_UNLIMITED_CACHE 0 +#endif +#ifndef ENABLE_ADAPTIVE_THREAD_CACHE +//! Enable adaptive thread cache size based on use heuristics +#define ENABLE_ADAPTIVE_THREAD_CACHE 0 +#endif +#ifndef DEFAULT_SPAN_MAP_COUNT +//! Default number of spans to map in call to map more virtual memory (default values yield 4MiB here) +#define DEFAULT_SPAN_MAP_COUNT 64 +#endif +#ifndef GLOBAL_CACHE_MULTIPLIER +//! Multiplier for global cache +#define GLOBAL_CACHE_MULTIPLIER 8 +#endif + +#if DISABLE_UNMAP && !ENABLE_GLOBAL_CACHE +#error Must use global cache if unmap is disabled +#endif + +#if DISABLE_UNMAP +#undef ENABLE_UNLIMITED_CACHE +#define ENABLE_UNLIMITED_CACHE 1 +#endif + +#if !ENABLE_GLOBAL_CACHE +#undef ENABLE_UNLIMITED_CACHE +#define ENABLE_UNLIMITED_CACHE 0 +#endif + +#if !ENABLE_THREAD_CACHE +#undef ENABLE_ADAPTIVE_THREAD_CACHE +#define ENABLE_ADAPTIVE_THREAD_CACHE 0 +#endif + +#if defined(_WIN32) || defined(__WIN32__) || defined(_WIN64) +# define PLATFORM_WINDOWS 1 +# define PLATFORM_POSIX 0 +#else +# define PLATFORM_WINDOWS 0 +# define PLATFORM_POSIX 1 +#endif + +/// Platform and arch specifics +#if defined(_MSC_VER) && !defined(__clang__) +# pragma warning (disable: 5105) +# ifndef FORCEINLINE +# define FORCEINLINE inline __forceinline +# endif +# define _Static_assert static_assert +#else +# ifndef FORCEINLINE +# define FORCEINLINE inline __attribute__((__always_inline__)) +# endif +#endif +#if PLATFORM_WINDOWS +# ifndef WIN32_LEAN_AND_MEAN +# define WIN32_LEAN_AND_MEAN +# endif +# include <windows.h> +# if ENABLE_VALIDATE_ARGS +# include <intsafe.h> +# endif +#else +# include <unistd.h> +# include <stdio.h> +# include <stdlib.h> +# include <time.h> +# if defined(__linux__) || defined(__ANDROID__) +# include <sys/prctl.h> +# if !defined(PR_SET_VMA) +# define PR_SET_VMA 0x53564d41 +# define PR_SET_VMA_ANON_NAME 0 +# endif +# endif +# if defined(__APPLE__) +# include <TargetConditionals.h> +# if !TARGET_OS_IPHONE && !TARGET_OS_SIMULATOR +# include <mach/mach_vm.h> +# include <mach/vm_statistics.h> +# endif +# include <pthread.h> +# endif +# if defined(__HAIKU__) || defined(__TINYC__) +# include <pthread.h> +# endif +#endif + +#include <stdint.h> +#include <string.h> +#include <errno.h> + +#if defined(_WIN32) && (!defined(BUILD_DYNAMIC_LINK) || !BUILD_DYNAMIC_LINK) +#include <fibersapi.h> +static DWORD fls_key; +#endif + +#if PLATFORM_POSIX +# include <sys/mman.h> +# include <sched.h> +# ifdef __FreeBSD__ +# include <sys/sysctl.h> +# define MAP_HUGETLB MAP_ALIGNED_SUPER +# ifndef PROT_MAX +# define PROT_MAX(f) 0 +# endif +# else +# define PROT_MAX(f) 0 +# endif +# ifdef __sun +extern int madvise(caddr_t, size_t, int); +# endif +# ifndef MAP_UNINITIALIZED +# define MAP_UNINITIALIZED 0 +# endif +#endif +#include <errno.h> + +#if ENABLE_ASSERTS +# undef NDEBUG +# if defined(_MSC_VER) && !defined(_DEBUG) +# define _DEBUG +# endif +# include <assert.h> +#define RPMALLOC_TOSTRING_M(x) #x +#define RPMALLOC_TOSTRING(x) RPMALLOC_TOSTRING_M(x) +#define rpmalloc_assert(truth, message) \ + do { \ + if (!(truth)) { \ + if (_memory_config.error_callback) { \ + _memory_config.error_callback( \ + message " (" RPMALLOC_TOSTRING(truth) ") at " __FILE__ ":" RPMALLOC_TOSTRING(__LINE__)); \ + } else { \ + assert((truth) && message); \ + } \ + } \ + } while (0) +#else +# define rpmalloc_assert(truth, message) do {} while(0) +#endif +#if ENABLE_STATISTICS +# include <stdio.h> +#endif + +////// +/// +/// Atomic access abstraction (since MSVC does not do C11 yet) +/// +////// + +#if defined(_MSC_VER) && !defined(__clang__) + +typedef volatile long atomic32_t; +typedef volatile long long atomic64_t; +typedef volatile void* atomicptr_t; + +static FORCEINLINE int32_t atomic_load32(atomic32_t* src) { return *src; } +static FORCEINLINE void atomic_store32(atomic32_t* dst, int32_t val) { *dst = val; } +static FORCEINLINE int32_t atomic_incr32(atomic32_t* val) { return (int32_t)InterlockedIncrement(val); } +static FORCEINLINE int32_t atomic_decr32(atomic32_t* val) { return (int32_t)InterlockedDecrement(val); } +static FORCEINLINE int32_t atomic_add32(atomic32_t* val, int32_t add) { return (int32_t)InterlockedExchangeAdd(val, add) + add; } +static FORCEINLINE int atomic_cas32_acquire(atomic32_t* dst, int32_t val, int32_t ref) { return (InterlockedCompareExchange(dst, val, ref) == ref) ? 1 : 0; } +static FORCEINLINE void atomic_store32_release(atomic32_t* dst, int32_t val) { *dst = val; } +static FORCEINLINE int64_t atomic_load64(atomic64_t* src) { return *src; } +static FORCEINLINE int64_t atomic_add64(atomic64_t* val, int64_t add) { return (int64_t)InterlockedExchangeAdd64(val, add) + add; } +static FORCEINLINE void* atomic_load_ptr(atomicptr_t* src) { return (void*)*src; } +static FORCEINLINE void atomic_store_ptr(atomicptr_t* dst, void* val) { *dst = val; } +static FORCEINLINE void atomic_store_ptr_release(atomicptr_t* dst, void* val) { *dst = val; } +static FORCEINLINE void* atomic_exchange_ptr_acquire(atomicptr_t* dst, void* val) { return (void*)InterlockedExchangePointer((void* volatile*)dst, val); } +static FORCEINLINE int atomic_cas_ptr(atomicptr_t* dst, void* val, void* ref) { return (InterlockedCompareExchangePointer((void* volatile*)dst, val, ref) == ref) ? 1 : 0; } + +#define EXPECTED(x) (x) +#define UNEXPECTED(x) (x) + +#else + +#include <stdatomic.h> + +typedef volatile _Atomic(int32_t) atomic32_t; +typedef volatile _Atomic(int64_t) atomic64_t; +typedef volatile _Atomic(void*) atomicptr_t; + +static FORCEINLINE int32_t atomic_load32(atomic32_t* src) { return atomic_load_explicit(src, memory_order_relaxed); } +static FORCEINLINE void atomic_store32(atomic32_t* dst, int32_t val) { atomic_store_explicit(dst, val, memory_order_relaxed); } +static FORCEINLINE int32_t atomic_incr32(atomic32_t* val) { return atomic_fetch_add_explicit(val, 1, memory_order_relaxed) + 1; } +static FORCEINLINE int32_t atomic_decr32(atomic32_t* val) { return atomic_fetch_add_explicit(val, -1, memory_order_relaxed) - 1; } +static FORCEINLINE int32_t atomic_add32(atomic32_t* val, int32_t add) { return atomic_fetch_add_explicit(val, add, memory_order_relaxed) + add; } +static FORCEINLINE int atomic_cas32_acquire(atomic32_t* dst, int32_t val, int32_t ref) { return atomic_compare_exchange_weak_explicit(dst, &ref, val, memory_order_acquire, memory_order_relaxed); } +static FORCEINLINE void atomic_store32_release(atomic32_t* dst, int32_t val) { atomic_store_explicit(dst, val, memory_order_release); } +static FORCEINLINE int64_t atomic_load64(atomic64_t* val) { return atomic_load_explicit(val, memory_order_relaxed); } +static FORCEINLINE int64_t atomic_add64(atomic64_t* val, int64_t add) { return atomic_fetch_add_explicit(val, add, memory_order_relaxed) + add; } +static FORCEINLINE void* atomic_load_ptr(atomicptr_t* src) { return atomic_load_explicit(src, memory_order_relaxed); } +static FORCEINLINE void atomic_store_ptr(atomicptr_t* dst, void* val) { atomic_store_explicit(dst, val, memory_order_relaxed); } +static FORCEINLINE void atomic_store_ptr_release(atomicptr_t* dst, void* val) { atomic_store_explicit(dst, val, memory_order_release); } +static FORCEINLINE void* atomic_exchange_ptr_acquire(atomicptr_t* dst, void* val) { return atomic_exchange_explicit(dst, val, memory_order_acquire); } +static FORCEINLINE int atomic_cas_ptr(atomicptr_t* dst, void* val, void* ref) { return atomic_compare_exchange_weak_explicit(dst, &ref, val, memory_order_relaxed, memory_order_relaxed); } + +#define EXPECTED(x) __builtin_expect((x), 1) +#define UNEXPECTED(x) __builtin_expect((x), 0) + +#endif + +//////////// +/// +/// Statistics related functions (evaluate to nothing when statistics not enabled) +/// +////// + +#if ENABLE_STATISTICS +# define _rpmalloc_stat_inc(counter) atomic_incr32(counter) +# define _rpmalloc_stat_dec(counter) atomic_decr32(counter) +# define _rpmalloc_stat_add(counter, value) atomic_add32(counter, (int32_t)(value)) +# define _rpmalloc_stat_add64(counter, value) atomic_add64(counter, (int64_t)(value)) +# define _rpmalloc_stat_add_peak(counter, value, peak) do { int32_t _cur_count = atomic_add32(counter, (int32_t)(value)); if (_cur_count > (peak)) peak = _cur_count; } while (0) +# define _rpmalloc_stat_sub(counter, value) atomic_add32(counter, -(int32_t)(value)) +# define _rpmalloc_stat_inc_alloc(heap, class_idx) do { \ + int32_t alloc_current = atomic_incr32(&heap->size_class_use[class_idx].alloc_current); \ + if (alloc_current > heap->size_class_use[class_idx].alloc_peak) \ + heap->size_class_use[class_idx].alloc_peak = alloc_current; \ + atomic_incr32(&heap->size_class_use[class_idx].alloc_total); \ +} while(0) +# define _rpmalloc_stat_inc_free(heap, class_idx) do { \ + atomic_decr32(&heap->size_class_use[class_idx].alloc_current); \ + atomic_incr32(&heap->size_class_use[class_idx].free_total); \ +} while(0) +#else +# define _rpmalloc_stat_inc(counter) do {} while(0) +# define _rpmalloc_stat_dec(counter) do {} while(0) +# define _rpmalloc_stat_add(counter, value) do {} while(0) +# define _rpmalloc_stat_add64(counter, value) do {} while(0) +# define _rpmalloc_stat_add_peak(counter, value, peak) do {} while (0) +# define _rpmalloc_stat_sub(counter, value) do {} while(0) +# define _rpmalloc_stat_inc_alloc(heap, class_idx) do {} while(0) +# define _rpmalloc_stat_inc_free(heap, class_idx) do {} while(0) +#endif + + +/// +/// Preconfigured limits and sizes +/// + +//! Granularity of a small allocation block (must be power of two) +#define SMALL_GRANULARITY 16 +//! Small granularity shift count +#define SMALL_GRANULARITY_SHIFT 4 +//! Number of small block size classes +#define SMALL_CLASS_COUNT 65 +//! Maximum size of a small block +#define SMALL_SIZE_LIMIT (SMALL_GRANULARITY * (SMALL_CLASS_COUNT - 1)) +//! Granularity of a medium allocation block +#define MEDIUM_GRANULARITY 512 +//! Medium granularity shift count +#define MEDIUM_GRANULARITY_SHIFT 9 +//! Number of medium block size classes +#define MEDIUM_CLASS_COUNT 61 +//! Total number of small + medium size classes +#define SIZE_CLASS_COUNT (SMALL_CLASS_COUNT + MEDIUM_CLASS_COUNT) +//! Number of large block size classes +#define LARGE_CLASS_COUNT 63 +//! Maximum size of a medium block +#define MEDIUM_SIZE_LIMIT (SMALL_SIZE_LIMIT + (MEDIUM_GRANULARITY * MEDIUM_CLASS_COUNT)) +//! Maximum size of a large block +#define LARGE_SIZE_LIMIT ((LARGE_CLASS_COUNT * _memory_span_size) - SPAN_HEADER_SIZE) +//! Size of a span header (must be a multiple of SMALL_GRANULARITY and a power of two) +#define SPAN_HEADER_SIZE 128 +//! Number of spans in thread cache +#define MAX_THREAD_SPAN_CACHE 400 +//! Number of spans to transfer between thread and global cache +#define THREAD_SPAN_CACHE_TRANSFER 64 +//! Number of spans in thread cache for large spans (must be greater than LARGE_CLASS_COUNT / 2) +#define MAX_THREAD_SPAN_LARGE_CACHE 100 +//! Number of spans to transfer between thread and global cache for large spans +#define THREAD_SPAN_LARGE_CACHE_TRANSFER 6 + +_Static_assert((SMALL_GRANULARITY & (SMALL_GRANULARITY - 1)) == 0, "Small granularity must be power of two"); +_Static_assert((SPAN_HEADER_SIZE & (SPAN_HEADER_SIZE - 1)) == 0, "Span header size must be power of two"); + +#if ENABLE_VALIDATE_ARGS +//! Maximum allocation size to avoid integer overflow +#undef MAX_ALLOC_SIZE +#define MAX_ALLOC_SIZE (((size_t)-1) - _memory_span_size) +#endif + +#define pointer_offset(ptr, ofs) (void*)((char*)(ptr) + (ptrdiff_t)(ofs)) +#define pointer_diff(first, second) (ptrdiff_t)((const char*)(first) - (const char*)(second)) + +#define INVALID_POINTER ((void*)((uintptr_t)-1)) + +#define SIZE_CLASS_LARGE SIZE_CLASS_COUNT +#define SIZE_CLASS_HUGE ((uint32_t)-1) + +//////////// +/// +/// Data types +/// +////// + +//! A memory heap, per thread +typedef struct heap_t heap_t; +//! Span of memory pages +typedef struct span_t span_t; +//! Span list +typedef struct span_list_t span_list_t; +//! Span active data +typedef struct span_active_t span_active_t; +//! Size class definition +typedef struct size_class_t size_class_t; +//! Global cache +typedef struct global_cache_t global_cache_t; + +//! Flag indicating span is the first (master) span of a split superspan +#define SPAN_FLAG_MASTER 1U +//! Flag indicating span is a secondary (sub) span of a split superspan +#define SPAN_FLAG_SUBSPAN 2U +//! Flag indicating span has blocks with increased alignment +#define SPAN_FLAG_ALIGNED_BLOCKS 4U +//! Flag indicating an unmapped master span +#define SPAN_FLAG_UNMAPPED_MASTER 8U + +#if ENABLE_ADAPTIVE_THREAD_CACHE || ENABLE_STATISTICS +struct span_use_t { + //! Current number of spans used (actually used, not in cache) + atomic32_t current; + //! High water mark of spans used + atomic32_t high; +#if ENABLE_STATISTICS + //! Number of spans in deferred list + atomic32_t spans_deferred; + //! Number of spans transitioned to global cache + atomic32_t spans_to_global; + //! Number of spans transitioned from global cache + atomic32_t spans_from_global; + //! Number of spans transitioned to thread cache + atomic32_t spans_to_cache; + //! Number of spans transitioned from thread cache + atomic32_t spans_from_cache; + //! Number of spans transitioned to reserved state + atomic32_t spans_to_reserved; + //! Number of spans transitioned from reserved state + atomic32_t spans_from_reserved; + //! Number of raw memory map calls + atomic32_t spans_map_calls; +#endif +}; +typedef struct span_use_t span_use_t; +#endif + +#if ENABLE_STATISTICS +struct size_class_use_t { + //! Current number of allocations + atomic32_t alloc_current; + //! Peak number of allocations + int32_t alloc_peak; + //! Total number of allocations + atomic32_t alloc_total; + //! Total number of frees + atomic32_t free_total; + //! Number of spans in use + atomic32_t spans_current; + //! Number of spans transitioned to cache + int32_t spans_peak; + //! Number of spans transitioned to cache + atomic32_t spans_to_cache; + //! Number of spans transitioned from cache + atomic32_t spans_from_cache; + //! Number of spans transitioned from reserved state + atomic32_t spans_from_reserved; + //! Number of spans mapped + atomic32_t spans_map_calls; + int32_t unused; +}; +typedef struct size_class_use_t size_class_use_t; +#endif + +// A span can either represent a single span of memory pages with size declared by span_map_count configuration variable, +// or a set of spans in a continuous region, a super span. Any reference to the term "span" usually refers to both a single +// span or a super span. A super span can further be divided into multiple spans (or this, super spans), where the first +// (super)span is the master and subsequent (super)spans are subspans. The master span keeps track of how many subspans +// that are still alive and mapped in virtual memory, and once all subspans and master have been unmapped the entire +// superspan region is released and unmapped (on Windows for example, the entire superspan range has to be released +// in the same call to release the virtual memory range, but individual subranges can be decommitted individually +// to reduce physical memory use). +struct span_t { + //! Free list + void* free_list; + //! Total block count of size class + uint32_t block_count; + //! Size class + uint32_t size_class; + //! Index of last block initialized in free list + uint32_t free_list_limit; + //! Number of used blocks remaining when in partial state + uint32_t used_count; + //! Deferred free list + atomicptr_t free_list_deferred; + //! Size of deferred free list, or list of spans when part of a cache list + uint32_t list_size; + //! Size of a block + uint32_t block_size; + //! Flags and counters + uint32_t flags; + //! Number of spans + uint32_t span_count; + //! Total span counter for master spans + uint32_t total_spans; + //! Offset from master span for subspans + uint32_t offset_from_master; + //! Remaining span counter, for master spans + atomic32_t remaining_spans; + //! Alignment offset + uint32_t align_offset; + //! Owning heap + heap_t* heap; + //! Next span + span_t* next; + //! Previous span + span_t* prev; +}; +_Static_assert(sizeof(span_t) <= SPAN_HEADER_SIZE, "span size mismatch"); + +struct span_cache_t { + size_t count; + span_t* span[MAX_THREAD_SPAN_CACHE]; +}; +typedef struct span_cache_t span_cache_t; + +struct span_large_cache_t { + size_t count; + span_t* span[MAX_THREAD_SPAN_LARGE_CACHE]; +}; +typedef struct span_large_cache_t span_large_cache_t; + +struct heap_size_class_t { + //! Free list of active span + void* free_list; + //! Double linked list of partially used spans with free blocks. + // Previous span pointer in head points to tail span of list. + span_t* partial_span; + //! Early level cache of fully free spans + span_t* cache; +}; +typedef struct heap_size_class_t heap_size_class_t; + +// Control structure for a heap, either a thread heap or a first class heap if enabled +struct heap_t { + //! Owning thread ID + uintptr_t owner_thread; + //! Free lists for each size class + heap_size_class_t size_class[SIZE_CLASS_COUNT]; +#if ENABLE_THREAD_CACHE + //! Arrays of fully freed spans, single span + span_cache_t span_cache; +#endif + //! List of deferred free spans (single linked list) + atomicptr_t span_free_deferred; + //! Number of full spans + size_t full_span_count; + //! Mapped but unused spans + span_t* span_reserve; + //! Master span for mapped but unused spans + span_t* span_reserve_master; + //! Number of mapped but unused spans + uint32_t spans_reserved; + //! Child count + atomic32_t child_count; + //! Next heap in id list + heap_t* next_heap; + //! Next heap in orphan list + heap_t* next_orphan; + //! Heap ID + int32_t id; + //! Finalization state flag + int finalize; + //! Master heap owning the memory pages + heap_t* master_heap; +#if ENABLE_THREAD_CACHE + //! Arrays of fully freed spans, large spans with > 1 span count + span_large_cache_t span_large_cache[LARGE_CLASS_COUNT - 1]; +#endif +#if RPMALLOC_FIRST_CLASS_HEAPS + //! Double linked list of fully utilized spans with free blocks for each size class. + // Previous span pointer in head points to tail span of list. + span_t* full_span[SIZE_CLASS_COUNT]; + //! Double linked list of large and huge spans allocated by this heap + span_t* large_huge_span; +#endif +#if ENABLE_ADAPTIVE_THREAD_CACHE || ENABLE_STATISTICS + //! Current and high water mark of spans used per span count + span_use_t span_use[LARGE_CLASS_COUNT]; +#endif +#if ENABLE_STATISTICS + //! Allocation stats per size class + size_class_use_t size_class_use[SIZE_CLASS_COUNT + 1]; + //! Number of bytes transitioned thread -> global + atomic64_t thread_to_global; + //! Number of bytes transitioned global -> thread + atomic64_t global_to_thread; +#endif +}; + +// Size class for defining a block size bucket +struct size_class_t { + //! Size of blocks in this class + uint32_t block_size; + //! Number of blocks in each chunk + uint16_t block_count; + //! Class index this class is merged with + uint16_t class_idx; +}; +_Static_assert(sizeof(size_class_t) == 8, "Size class size mismatch"); + +struct global_cache_t { + //! Cache lock + atomic32_t lock; + //! Cache count + uint32_t count; +#if ENABLE_STATISTICS + //! Insert count + size_t insert_count; + //! Extract count + size_t extract_count; +#endif + //! Cached spans + span_t* span[GLOBAL_CACHE_MULTIPLIER * MAX_THREAD_SPAN_CACHE]; + //! Unlimited cache overflow + span_t* overflow; +}; + +//////////// +/// +/// Global data +/// +////// + +//! Default span size (64KiB) +#define _memory_default_span_size (64 * 1024) +#define _memory_default_span_size_shift 16 +#define _memory_default_span_mask (~((uintptr_t)(_memory_span_size - 1))) + +//! Initialized flag +static int _rpmalloc_initialized; +//! Main thread ID +static uintptr_t _rpmalloc_main_thread_id; +//! Configuration +static rpmalloc_config_t _memory_config; +//! Memory page size +static size_t _memory_page_size; +//! Shift to divide by page size +static size_t _memory_page_size_shift; +//! Granularity at which memory pages are mapped by OS +static size_t _memory_map_granularity; +#if RPMALLOC_CONFIGURABLE +//! Size of a span of memory pages +static size_t _memory_span_size; +//! Shift to divide by span size +static size_t _memory_span_size_shift; +//! Mask to get to start of a memory span +static uintptr_t _memory_span_mask; +#else +//! Hardwired span size +#define _memory_span_size _memory_default_span_size +#define _memory_span_size_shift _memory_default_span_size_shift +#define _memory_span_mask _memory_default_span_mask +#endif +//! Number of spans to map in each map call +static size_t _memory_span_map_count; +//! Number of spans to keep reserved in each heap +static size_t _memory_heap_reserve_count; +//! Global size classes +static size_class_t _memory_size_class[SIZE_CLASS_COUNT]; +//! Run-time size limit of medium blocks +static size_t _memory_medium_size_limit; +//! Heap ID counter +static atomic32_t _memory_heap_id; +//! Huge page support +static int _memory_huge_pages; +#if ENABLE_GLOBAL_CACHE +//! Global span cache +static global_cache_t _memory_span_cache[LARGE_CLASS_COUNT]; +#endif +//! Global reserved spans +static span_t* _memory_global_reserve; +//! Global reserved count +static size_t _memory_global_reserve_count; +//! Global reserved master +static span_t* _memory_global_reserve_master; +//! All heaps +static heap_t* _memory_heaps[HEAP_ARRAY_SIZE]; +//! Used to restrict access to mapping memory for huge pages +static atomic32_t _memory_global_lock; +//! Orphaned heaps +static heap_t* _memory_orphan_heaps; +#if RPMALLOC_FIRST_CLASS_HEAPS +//! Orphaned heaps (first class heaps) +static heap_t* _memory_first_class_orphan_heaps; +#endif +#if ENABLE_STATISTICS +//! Allocations counter +static atomic64_t _allocation_counter; +//! Deallocations counter +static atomic64_t _deallocation_counter; +//! Active heap count +static atomic32_t _memory_active_heaps; +//! Number of currently mapped memory pages +static atomic32_t _mapped_pages; +//! Peak number of concurrently mapped memory pages +static int32_t _mapped_pages_peak; +//! Number of mapped master spans +static atomic32_t _master_spans; +//! Number of unmapped dangling master spans +static atomic32_t _unmapped_master_spans; +//! Running counter of total number of mapped memory pages since start +static atomic32_t _mapped_total; +//! Running counter of total number of unmapped memory pages since start +static atomic32_t _unmapped_total; +//! Number of currently mapped memory pages in OS calls +static atomic32_t _mapped_pages_os; +//! Number of currently allocated pages in huge allocations +static atomic32_t _huge_pages_current; +//! Peak number of currently allocated pages in huge allocations +static int32_t _huge_pages_peak; +#endif + +//////////// +/// +/// Thread local heap and ID +/// +////// + +//! Current thread heap +#if ((defined(__APPLE__) || defined(__HAIKU__)) && ENABLE_PRELOAD) || defined(__TINYC__) +static pthread_key_t _memory_thread_heap; +#else +# ifdef _MSC_VER +# define _Thread_local __declspec(thread) +# define TLS_MODEL +# else +# ifndef __HAIKU__ +# define TLS_MODEL __attribute__((tls_model("initial-exec"))) +# else +# define TLS_MODEL +# endif +# if !defined(__clang__) && defined(__GNUC__) +# define _Thread_local __thread +# endif +# endif +static _Thread_local heap_t* _memory_thread_heap TLS_MODEL; +#endif + +static inline heap_t* +get_thread_heap_raw(void) { +#if (defined(__APPLE__) || defined(__HAIKU__)) && ENABLE_PRELOAD + return pthread_getspecific(_memory_thread_heap); +#else + return _memory_thread_heap; +#endif +} + +//! Get the current thread heap +static inline heap_t* +get_thread_heap(void) { + heap_t* heap = get_thread_heap_raw(); +#if ENABLE_PRELOAD + if (EXPECTED(heap != 0)) + return heap; + rpmalloc_initialize(); + return get_thread_heap_raw(); +#else + return heap; +#endif +} + +//! Fast thread ID +static inline uintptr_t +get_thread_id(void) { +#if defined(_WIN32) + return (uintptr_t)((void*)NtCurrentTeb()); +#elif (defined(__GNUC__) || defined(__clang__)) && !defined(__CYGWIN__) + uintptr_t tid; +# if defined(__i386__) + __asm__("movl %%gs:0, %0" : "=r" (tid) : : ); +# elif defined(__x86_64__) +# if defined(__MACH__) + __asm__("movq %%gs:0, %0" : "=r" (tid) : : ); +# else + __asm__("movq %%fs:0, %0" : "=r" (tid) : : ); +# endif +# elif defined(__arm__) + __asm__ volatile ("mrc p15, 0, %0, c13, c0, 3" : "=r" (tid)); +# elif defined(__aarch64__) +# if defined(__MACH__) + // tpidr_el0 likely unused, always return 0 on iOS + __asm__ volatile ("mrs %0, tpidrro_el0" : "=r" (tid)); +# else + __asm__ volatile ("mrs %0, tpidr_el0" : "=r" (tid)); +# endif +# else + tid = (uintptr_t)((void*)get_thread_heap_raw()); +# endif + return tid; +#else + return (uintptr_t)((void*)get_thread_heap_raw()); +#endif +} + +//! Set the current thread heap +static void +set_thread_heap(heap_t* heap) { +#if ((defined(__APPLE__) || defined(__HAIKU__)) && ENABLE_PRELOAD) || defined(__TINYC__) + pthread_setspecific(_memory_thread_heap, heap); +#else + _memory_thread_heap = heap; +#endif + if (heap) + heap->owner_thread = get_thread_id(); +} + +//! Set main thread ID +extern void +rpmalloc_set_main_thread(void); + +void +rpmalloc_set_main_thread(void) { + _rpmalloc_main_thread_id = get_thread_id(); +} + +static void +_rpmalloc_spin(void) { +#if defined(_MSC_VER) + _mm_pause(); +#elif defined(__x86_64__) || defined(__i386__) + __asm__ volatile("pause" ::: "memory"); +#elif defined(__aarch64__) || (defined(__arm__) && __ARM_ARCH >= 7) + __asm__ volatile("yield" ::: "memory"); +#elif defined(__powerpc__) || defined(__powerpc64__) + // No idea if ever been compiled in such archs but ... as precaution + __asm__ volatile("or 27,27,27"); +#elif defined(__sparc__) + __asm__ volatile("rd %ccr, %g0 \n\trd %ccr, %g0 \n\trd %ccr, %g0"); +#else + struct timespec ts = {0}; + nanosleep(&ts, 0); +#endif +} + +#if defined(_WIN32) && (!defined(BUILD_DYNAMIC_LINK) || !BUILD_DYNAMIC_LINK) +static void NTAPI +_rpmalloc_thread_destructor(void* value) { +#if ENABLE_OVERRIDE + // If this is called on main thread it means rpmalloc_finalize + // has not been called and shutdown is forced (through _exit) or unclean + if (get_thread_id() == _rpmalloc_main_thread_id) + return; +#endif + if (value) + rpmalloc_thread_finalize(1); +} +#endif + + +//////////// +/// +/// Low level memory map/unmap +/// +////// + +static void +_rpmalloc_set_name(void* address, size_t size) { +#if defined(__linux__) || defined(__ANDROID__) + const char *name = _memory_huge_pages ? _memory_config.huge_page_name : _memory_config.page_name; + if (address == MAP_FAILED || !name) + return; + // If the kernel does not support CONFIG_ANON_VMA_NAME or if the call fails + // (e.g. invalid name) it is a no-op basically. + (void)prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, (uintptr_t)address, size, (uintptr_t)name); +#else + (void)sizeof(size); + (void)sizeof(address); +#endif +} + + +//! Map more virtual memory +// size is number of bytes to map +// offset receives the offset in bytes from start of mapped region +// returns address to start of mapped region to use +static void* +_rpmalloc_mmap(size_t size, size_t* offset) { + rpmalloc_assert(!(size % _memory_page_size), "Invalid mmap size"); + rpmalloc_assert(size >= _memory_page_size, "Invalid mmap size"); + void* address = _memory_config.memory_map(size, offset); + if (EXPECTED(address != 0)) { + _rpmalloc_stat_add_peak(&_mapped_pages, (size >> _memory_page_size_shift), _mapped_pages_peak); + _rpmalloc_stat_add(&_mapped_total, (size >> _memory_page_size_shift)); + } + return address; +} + +//! Unmap virtual memory +// address is the memory address to unmap, as returned from _memory_map +// size is the number of bytes to unmap, which might be less than full region for a partial unmap +// offset is the offset in bytes to the actual mapped region, as set by _memory_map +// release is set to 0 for partial unmap, or size of entire range for a full unmap +static void +_rpmalloc_unmap(void* address, size_t size, size_t offset, size_t release) { + rpmalloc_assert(!release || (release >= size), "Invalid unmap size"); + rpmalloc_assert(!release || (release >= _memory_page_size), "Invalid unmap size"); + if (release) { + rpmalloc_assert(!(release % _memory_page_size), "Invalid unmap size"); + _rpmalloc_stat_sub(&_mapped_pages, (release >> _memory_page_size_shift)); + _rpmalloc_stat_add(&_unmapped_total, (release >> _memory_page_size_shift)); + } + _memory_config.memory_unmap(address, size, offset, release); +} + +//! Default implementation to map new pages to virtual memory +static void* +_rpmalloc_mmap_os(size_t size, size_t* offset) { + //Either size is a heap (a single page) or a (multiple) span - we only need to align spans, and only if larger than map granularity + size_t padding = ((size >= _memory_span_size) && (_memory_span_size > _memory_map_granularity)) ? _memory_span_size : 0; + rpmalloc_assert(size >= _memory_page_size, "Invalid mmap size"); +#if PLATFORM_WINDOWS + //Ok to MEM_COMMIT - according to MSDN, "actual physical pages are not allocated unless/until the virtual addresses are actually accessed" + void* ptr = VirtualAlloc(0, size + padding, (_memory_huge_pages ? MEM_LARGE_PAGES : 0) | MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); + if (!ptr) { + if (_memory_config.map_fail_callback) { + if (_memory_config.map_fail_callback(size + padding)) + return _rpmalloc_mmap_os(size, offset); + } else { + rpmalloc_assert(ptr, "Failed to map virtual memory block"); + } + return 0; + } +#else + int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_UNINITIALIZED; +# if defined(__APPLE__) && !TARGET_OS_IPHONE && !TARGET_OS_SIMULATOR + int fd = (int)VM_MAKE_TAG(240U); + if (_memory_huge_pages) + fd |= VM_FLAGS_SUPERPAGE_SIZE_2MB; + void* ptr = mmap(0, size + padding, PROT_READ | PROT_WRITE, flags, fd, 0); +# elif defined(MAP_HUGETLB) + void* ptr = mmap(0, size + padding, PROT_READ | PROT_WRITE | PROT_MAX(PROT_READ | PROT_WRITE), (_memory_huge_pages ? MAP_HUGETLB : 0) | flags, -1, 0); +# if defined(MADV_HUGEPAGE) + // In some configurations, huge pages allocations might fail thus + // we fallback to normal allocations and promote the region as transparent huge page + if ((ptr == MAP_FAILED || !ptr) && _memory_huge_pages) { + ptr = mmap(0, size + padding, PROT_READ | PROT_WRITE, flags, -1, 0); + if (ptr && ptr != MAP_FAILED) { + int prm = madvise(ptr, size + padding, MADV_HUGEPAGE); + (void)prm; + rpmalloc_assert((prm == 0), "Failed to promote the page to THP"); + } + } +# endif + _rpmalloc_set_name(ptr, size + padding); +# elif defined(MAP_ALIGNED) + const size_t align = (sizeof(size_t) * 8) - (size_t)(__builtin_clzl(size - 1)); + void* ptr = mmap(0, size + padding, PROT_READ | PROT_WRITE, (_memory_huge_pages ? MAP_ALIGNED(align) : 0) | flags, -1, 0); +# elif defined(MAP_ALIGN) + caddr_t base = (_memory_huge_pages ? (caddr_t)(4 << 20) : 0); + void* ptr = mmap(base, size + padding, PROT_READ | PROT_WRITE, (_memory_huge_pages ? MAP_ALIGN : 0) | flags, -1, 0); +# else + void* ptr = mmap(0, size + padding, PROT_READ | PROT_WRITE, flags, -1, 0); +# endif + if ((ptr == MAP_FAILED) || !ptr) { + if (_memory_config.map_fail_callback) { + if (_memory_config.map_fail_callback(size + padding)) + return _rpmalloc_mmap_os(size, offset); + } else if (errno != ENOMEM) { + rpmalloc_assert((ptr != MAP_FAILED) && ptr, "Failed to map virtual memory block"); + } + return 0; + } +#endif + _rpmalloc_stat_add(&_mapped_pages_os, (int32_t)((size + padding) >> _memory_page_size_shift)); + if (padding) { + size_t final_padding = padding - ((uintptr_t)ptr & ~_memory_span_mask); + rpmalloc_assert(final_padding <= _memory_span_size, "Internal failure in padding"); + rpmalloc_assert(final_padding <= padding, "Internal failure in padding"); + rpmalloc_assert(!(final_padding % 8), "Internal failure in padding"); + ptr = pointer_offset(ptr, final_padding); + *offset = final_padding >> 3; + } + rpmalloc_assert((size < _memory_span_size) || !((uintptr_t)ptr & ~_memory_span_mask), "Internal failure in padding"); + return ptr; +} + +//! Default implementation to unmap pages from virtual memory +static void +_rpmalloc_unmap_os(void* address, size_t size, size_t offset, size_t release) { + rpmalloc_assert(release || (offset == 0), "Invalid unmap size"); + rpmalloc_assert(!release || (release >= _memory_page_size), "Invalid unmap size"); + rpmalloc_assert(size >= _memory_page_size, "Invalid unmap size"); + if (release && offset) { + offset <<= 3; + address = pointer_offset(address, -(int32_t)offset); + if ((release >= _memory_span_size) && (_memory_span_size > _memory_map_granularity)) { + //Padding is always one span size + release += _memory_span_size; + } + } +#if !DISABLE_UNMAP +#if PLATFORM_WINDOWS + if (!VirtualFree(address, release ? 0 : size, release ? MEM_RELEASE : MEM_DECOMMIT)) { + rpmalloc_assert(0, "Failed to unmap virtual memory block"); + } +#else + if (release) { + if (munmap(address, release)) { + rpmalloc_assert(0, "Failed to unmap virtual memory block"); + } + } else { +#if defined(MADV_FREE_REUSABLE) + int ret; + while ((ret = madvise(address, size, MADV_FREE_REUSABLE)) == -1 && (errno == EAGAIN)) + errno = 0; + if ((ret == -1) && (errno != 0)) { +#elif defined(MADV_DONTNEED) + if (madvise(address, size, MADV_DONTNEED)) { +#elif defined(MADV_PAGEOUT) + if (madvise(address, size, MADV_PAGEOUT)) { +#elif defined(MADV_FREE) + if (madvise(address, size, MADV_FREE)) { +#else + if (posix_madvise(address, size, POSIX_MADV_DONTNEED)) { +#endif + rpmalloc_assert(0, "Failed to madvise virtual memory block as free"); + } + } +#endif +#endif + if (release) + _rpmalloc_stat_sub(&_mapped_pages_os, release >> _memory_page_size_shift); +} + +static void +_rpmalloc_span_mark_as_subspan_unless_master(span_t* master, span_t* subspan, size_t span_count); + +//! Use global reserved spans to fulfill a memory map request (reserve size must be checked by caller) +static span_t* +_rpmalloc_global_get_reserved_spans(size_t span_count) { + span_t* span = _memory_global_reserve; + _rpmalloc_span_mark_as_subspan_unless_master(_memory_global_reserve_master, span, span_count); + _memory_global_reserve_count -= span_count; + if (_memory_global_reserve_count) + _memory_global_reserve = (span_t*)pointer_offset(span, span_count << _memory_span_size_shift); + else + _memory_global_reserve = 0; + return span; +} + +//! Store the given spans as global reserve (must only be called from within new heap allocation, not thread safe) +static void +_rpmalloc_global_set_reserved_spans(span_t* master, span_t* reserve, size_t reserve_span_count) { + _memory_global_reserve_master = master; + _memory_global_reserve_count = reserve_span_count; + _memory_global_reserve = reserve; +} + + +//////////// +/// +/// Span linked list management +/// +////// + +//! Add a span to double linked list at the head +static void +_rpmalloc_span_double_link_list_add(span_t** head, span_t* span) { + if (*head) + (*head)->prev = span; + span->next = *head; + *head = span; +} + +//! Pop head span from double linked list +static void +_rpmalloc_span_double_link_list_pop_head(span_t** head, span_t* span) { + rpmalloc_assert(*head == span, "Linked list corrupted"); + span = *head; + *head = span->next; +} + +//! Remove a span from double linked list +static void +_rpmalloc_span_double_link_list_remove(span_t** head, span_t* span) { + rpmalloc_assert(*head, "Linked list corrupted"); + if (*head == span) { + *head = span->next; + } else { + span_t* next_span = span->next; + span_t* prev_span = span->prev; + prev_span->next = next_span; + if (EXPECTED(next_span != 0)) + next_span->prev = prev_span; + } +} + + +//////////// +/// +/// Span control +/// +////// + +static void +_rpmalloc_heap_cache_insert(heap_t* heap, span_t* span); + +static void +_rpmalloc_heap_finalize(heap_t* heap); + +static void +_rpmalloc_heap_set_reserved_spans(heap_t* heap, span_t* master, span_t* reserve, size_t reserve_span_count); + +//! Declare the span to be a subspan and store distance from master span and span count +static void +_rpmalloc_span_mark_as_subspan_unless_master(span_t* master, span_t* subspan, size_t span_count) { + rpmalloc_assert((subspan != master) || (subspan->flags & SPAN_FLAG_MASTER), "Span master pointer and/or flag mismatch"); + if (subspan != master) { + subspan->flags = SPAN_FLAG_SUBSPAN; + subspan->offset_from_master = (uint32_t)((uintptr_t)pointer_diff(subspan, master) >> _memory_span_size_shift); + subspan->align_offset = 0; + } + subspan->span_count = (uint32_t)span_count; +} + +//! Use reserved spans to fulfill a memory map request (reserve size must be checked by caller) +static span_t* +_rpmalloc_span_map_from_reserve(heap_t* heap, size_t span_count) { + //Update the heap span reserve + span_t* span = heap->span_reserve; + heap->span_reserve = (span_t*)pointer_offset(span, span_count * _memory_span_size); + heap->spans_reserved -= (uint32_t)span_count; + + _rpmalloc_span_mark_as_subspan_unless_master(heap->span_reserve_master, span, span_count); + if (span_count <= LARGE_CLASS_COUNT) + _rpmalloc_stat_inc(&heap->span_use[span_count - 1].spans_from_reserved); + + return span; +} + +//! Get the aligned number of spans to map in based on wanted count, configured mapping granularity and the page size +static size_t +_rpmalloc_span_align_count(size_t span_count) { + size_t request_count = (span_count > _memory_span_map_count) ? span_count : _memory_span_map_count; + if ((_memory_page_size > _memory_span_size) && ((request_count * _memory_span_size) % _memory_page_size)) + request_count += _memory_span_map_count - (request_count % _memory_span_map_count); + return request_count; +} + +//! Setup a newly mapped span +static void +_rpmalloc_span_initialize(span_t* span, size_t total_span_count, size_t span_count, size_t align_offset) { + span->total_spans = (uint32_t)total_span_count; + span->span_count = (uint32_t)span_count; + span->align_offset = (uint32_t)align_offset; + span->flags = SPAN_FLAG_MASTER; + atomic_store32(&span->remaining_spans, (int32_t)total_span_count); +} + +static void +_rpmalloc_span_unmap(span_t* span); + +//! Map an aligned set of spans, taking configured mapping granularity and the page size into account +static span_t* +_rpmalloc_span_map_aligned_count(heap_t* heap, size_t span_count) { + //If we already have some, but not enough, reserved spans, release those to heap cache and map a new + //full set of spans. Otherwise we would waste memory if page size > span size (huge pages) + size_t aligned_span_count = _rpmalloc_span_align_count(span_count); + size_t align_offset = 0; + span_t* span = (span_t*)_rpmalloc_mmap(aligned_span_count * _memory_span_size, &align_offset); + if (!span) + return 0; + _rpmalloc_span_initialize(span, aligned_span_count, span_count, align_offset); + _rpmalloc_stat_inc(&_master_spans); + if (span_count <= LARGE_CLASS_COUNT) + _rpmalloc_stat_inc(&heap->span_use[span_count - 1].spans_map_calls); + if (aligned_span_count > span_count) { + span_t* reserved_spans = (span_t*)pointer_offset(span, span_count * _memory_span_size); + size_t reserved_count = aligned_span_count - span_count; + if (heap->spans_reserved) { + _rpmalloc_span_mark_as_subspan_unless_master(heap->span_reserve_master, heap->span_reserve, heap->spans_reserved); + _rpmalloc_heap_cache_insert(heap, heap->span_reserve); + } + if (reserved_count > _memory_heap_reserve_count) { + // If huge pages or eager spam map count, the global reserve spin lock is held by caller, _rpmalloc_span_map + rpmalloc_assert(atomic_load32(&_memory_global_lock) == 1, "Global spin lock not held as expected"); + size_t remain_count = reserved_count - _memory_heap_reserve_count; + reserved_count = _memory_heap_reserve_count; + span_t* remain_span = (span_t*)pointer_offset(reserved_spans, reserved_count * _memory_span_size); + if (_memory_global_reserve) { + _rpmalloc_span_mark_as_subspan_unless_master(_memory_global_reserve_master, _memory_global_reserve, _memory_global_reserve_count); + _rpmalloc_span_unmap(_memory_global_reserve); + } + _rpmalloc_global_set_reserved_spans(span, remain_span, remain_count); + } + _rpmalloc_heap_set_reserved_spans(heap, span, reserved_spans, reserved_count); + } + return span; +} + +//! Map in memory pages for the given number of spans (or use previously reserved pages) +static span_t* +_rpmalloc_span_map(heap_t* heap, size_t span_count) { + if (span_count <= heap->spans_reserved) + return _rpmalloc_span_map_from_reserve(heap, span_count); + span_t* span = 0; + int use_global_reserve = (_memory_page_size > _memory_span_size) || (_memory_span_map_count > _memory_heap_reserve_count); + if (use_global_reserve) { + // If huge pages, make sure only one thread maps more memory to avoid bloat + while (!atomic_cas32_acquire(&_memory_global_lock, 1, 0)) + _rpmalloc_spin(); + if (_memory_global_reserve_count >= span_count) { + size_t reserve_count = (!heap->spans_reserved ? _memory_heap_reserve_count : span_count); + if (_memory_global_reserve_count < reserve_count) + reserve_count = _memory_global_reserve_count; + span = _rpmalloc_global_get_reserved_spans(reserve_count); + if (span) { + if (reserve_count > span_count) { + span_t* reserved_span = (span_t*)pointer_offset(span, span_count << _memory_span_size_shift); + _rpmalloc_heap_set_reserved_spans(heap, _memory_global_reserve_master, reserved_span, reserve_count - span_count); + } + // Already marked as subspan in _rpmalloc_global_get_reserved_spans + span->span_count = (uint32_t)span_count; + } + } + } + if (!span) + span = _rpmalloc_span_map_aligned_count(heap, span_count); + if (use_global_reserve) + atomic_store32_release(&_memory_global_lock, 0); + return span; +} + +//! Unmap memory pages for the given number of spans (or mark as unused if no partial unmappings) +static void +_rpmalloc_span_unmap(span_t* span) { + rpmalloc_assert((span->flags & SPAN_FLAG_MASTER) || (span->flags & SPAN_FLAG_SUBSPAN), "Span flag corrupted"); + rpmalloc_assert(!(span->flags & SPAN_FLAG_MASTER) || !(span->flags & SPAN_FLAG_SUBSPAN), "Span flag corrupted"); + + int is_master = !!(span->flags & SPAN_FLAG_MASTER); + span_t* master = is_master ? span : ((span_t*)pointer_offset(span, -(intptr_t)((uintptr_t)span->offset_from_master * _memory_span_size))); + rpmalloc_assert(is_master || (span->flags & SPAN_FLAG_SUBSPAN), "Span flag corrupted"); + rpmalloc_assert(master->flags & SPAN_FLAG_MASTER, "Span flag corrupted"); + + size_t span_count = span->span_count; + if (!is_master) { + //Directly unmap subspans (unless huge pages, in which case we defer and unmap entire page range with master) + rpmalloc_assert(span->align_offset == 0, "Span align offset corrupted"); + if (_memory_span_size >= _memory_page_size) + _rpmalloc_unmap(span, span_count * _memory_span_size, 0, 0); + } else { + //Special double flag to denote an unmapped master + //It must be kept in memory since span header must be used + span->flags |= SPAN_FLAG_MASTER | SPAN_FLAG_SUBSPAN | SPAN_FLAG_UNMAPPED_MASTER; + _rpmalloc_stat_add(&_unmapped_master_spans, 1); + } + + if (atomic_add32(&master->remaining_spans, -(int32_t)span_count) <= 0) { + //Everything unmapped, unmap the master span with release flag to unmap the entire range of the super span + rpmalloc_assert(!!(master->flags & SPAN_FLAG_MASTER) && !!(master->flags & SPAN_FLAG_SUBSPAN), "Span flag corrupted"); + size_t unmap_count = master->span_count; + if (_memory_span_size < _memory_page_size) + unmap_count = master->total_spans; + _rpmalloc_stat_sub(&_master_spans, 1); + _rpmalloc_stat_sub(&_unmapped_master_spans, 1); + _rpmalloc_unmap(master, unmap_count * _memory_span_size, master->align_offset, (size_t)master->total_spans * _memory_span_size); + } +} + +//! Move the span (used for small or medium allocations) to the heap thread cache +static void +_rpmalloc_span_release_to_cache(heap_t* heap, span_t* span) { + rpmalloc_assert(heap == span->heap, "Span heap pointer corrupted"); + rpmalloc_assert(span->size_class < SIZE_CLASS_COUNT, "Invalid span size class"); + rpmalloc_assert(span->span_count == 1, "Invalid span count"); +#if ENABLE_ADAPTIVE_THREAD_CACHE || ENABLE_STATISTICS + atomic_decr32(&heap->span_use[0].current); +#endif + _rpmalloc_stat_dec(&heap->size_class_use[span->size_class].spans_current); + if (!heap->finalize) { + _rpmalloc_stat_inc(&heap->span_use[0].spans_to_cache); + _rpmalloc_stat_inc(&heap->size_class_use[span->size_class].spans_to_cache); + if (heap->size_class[span->size_class].cache) + _rpmalloc_heap_cache_insert(heap, heap->size_class[span->size_class].cache); + heap->size_class[span->size_class].cache = span; + } else { + _rpmalloc_span_unmap(span); + } +} + +//! Initialize a (partial) free list up to next system memory page, while reserving the first block +//! as allocated, returning number of blocks in list +static uint32_t +free_list_partial_init(void** list, void** first_block, void* page_start, void* block_start, uint32_t block_count, uint32_t block_size) { + rpmalloc_assert(block_count, "Internal failure"); + *first_block = block_start; + if (block_count > 1) { + void* free_block = pointer_offset(block_start, block_size); + void* block_end = pointer_offset(block_start, (size_t)block_size * block_count); + //If block size is less than half a memory page, bound init to next memory page boundary + if (block_size < (_memory_page_size >> 1)) { + void* page_end = pointer_offset(page_start, _memory_page_size); + if (page_end < block_end) + block_end = page_end; + } + *list = free_block; + block_count = 2; + void* next_block = pointer_offset(free_block, block_size); + while (next_block < block_end) { + *((void**)free_block) = next_block; + free_block = next_block; + ++block_count; + next_block = pointer_offset(next_block, block_size); + } + *((void**)free_block) = 0; + } else { + *list = 0; + } + return block_count; +} + +//! Initialize an unused span (from cache or mapped) to be new active span, putting the initial free list in heap class free list +static void* +_rpmalloc_span_initialize_new(heap_t* heap, heap_size_class_t* heap_size_class, span_t* span, uint32_t class_idx) { + rpmalloc_assert(span->span_count == 1, "Internal failure"); + size_class_t* size_class = _memory_size_class + class_idx; + span->size_class = class_idx; + span->heap = heap; + span->flags &= ~SPAN_FLAG_ALIGNED_BLOCKS; + span->block_size = size_class->block_size; + span->block_count = size_class->block_count; + span->free_list = 0; + span->list_size = 0; + atomic_store_ptr_release(&span->free_list_deferred, 0); + + //Setup free list. Only initialize one system page worth of free blocks in list + void* block; + span->free_list_limit = free_list_partial_init(&heap_size_class->free_list, &block, + span, pointer_offset(span, SPAN_HEADER_SIZE), size_class->block_count, size_class->block_size); + //Link span as partial if there remains blocks to be initialized as free list, or full if fully initialized + if (span->free_list_limit < span->block_count) { + _rpmalloc_span_double_link_list_add(&heap_size_class->partial_span, span); + span->used_count = span->free_list_limit; + } else { +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_add(&heap->full_span[class_idx], span); +#endif + ++heap->full_span_count; + span->used_count = span->block_count; + } + return block; +} + +static void +_rpmalloc_span_extract_free_list_deferred(span_t* span) { + // We need acquire semantics on the CAS operation since we are interested in the list size + // Refer to _rpmalloc_deallocate_defer_small_or_medium for further comments on this dependency + do { + span->free_list = atomic_exchange_ptr_acquire(&span->free_list_deferred, INVALID_POINTER); + } while (span->free_list == INVALID_POINTER); + span->used_count -= span->list_size; + span->list_size = 0; + atomic_store_ptr_release(&span->free_list_deferred, 0); +} + +static int +_rpmalloc_span_is_fully_utilized(span_t* span) { + rpmalloc_assert(span->free_list_limit <= span->block_count, "Span free list corrupted"); + return !span->free_list && (span->free_list_limit >= span->block_count); +} + +static int +_rpmalloc_span_finalize(heap_t* heap, size_t iclass, span_t* span, span_t** list_head) { + void* free_list = heap->size_class[iclass].free_list; + span_t* class_span = (span_t*)((uintptr_t)free_list & _memory_span_mask); + if (span == class_span) { + // Adopt the heap class free list back into the span free list + void* block = span->free_list; + void* last_block = 0; + while (block) { + last_block = block; + block = *((void**)block); + } + uint32_t free_count = 0; + block = free_list; + while (block) { + ++free_count; + block = *((void**)block); + } + if (last_block) { + *((void**)last_block) = free_list; + } else { + span->free_list = free_list; + } + heap->size_class[iclass].free_list = 0; + span->used_count -= free_count; + } + //If this assert triggers you have memory leaks + rpmalloc_assert(span->list_size == span->used_count, "Memory leak detected"); + if (span->list_size == span->used_count) { + _rpmalloc_stat_dec(&heap->span_use[0].current); + _rpmalloc_stat_dec(&heap->size_class_use[iclass].spans_current); + // This function only used for spans in double linked lists + if (list_head) + _rpmalloc_span_double_link_list_remove(list_head, span); + _rpmalloc_span_unmap(span); + return 1; + } + return 0; +} + + +//////////// +/// +/// Global cache +/// +////// + +#if ENABLE_GLOBAL_CACHE + +//! Finalize a global cache +static void +_rpmalloc_global_cache_finalize(global_cache_t* cache) { + while (!atomic_cas32_acquire(&cache->lock, 1, 0)) + _rpmalloc_spin(); + + for (size_t ispan = 0; ispan < cache->count; ++ispan) + _rpmalloc_span_unmap(cache->span[ispan]); + cache->count = 0; + + while (cache->overflow) { + span_t* span = cache->overflow; + cache->overflow = span->next; + _rpmalloc_span_unmap(span); + } + + atomic_store32_release(&cache->lock, 0); +} + +static void +_rpmalloc_global_cache_insert_spans(span_t** span, size_t span_count, size_t count) { + const size_t cache_limit = (span_count == 1) ? + GLOBAL_CACHE_MULTIPLIER * MAX_THREAD_SPAN_CACHE : + GLOBAL_CACHE_MULTIPLIER * (MAX_THREAD_SPAN_LARGE_CACHE - (span_count >> 1)); + + global_cache_t* cache = &_memory_span_cache[span_count - 1]; + + size_t insert_count = count; + while (!atomic_cas32_acquire(&cache->lock, 1, 0)) + _rpmalloc_spin(); + +#if ENABLE_STATISTICS + cache->insert_count += count; +#endif + if ((cache->count + insert_count) > cache_limit) + insert_count = cache_limit - cache->count; + + memcpy(cache->span + cache->count, span, sizeof(span_t*) * insert_count); + cache->count += (uint32_t)insert_count; + +#if ENABLE_UNLIMITED_CACHE + while (insert_count < count) { +#else + // Enable unlimited cache if huge pages, or we will leak since it is unlikely that an entire huge page + // will be unmapped, and we're unable to partially decommit a huge page + while ((_memory_page_size > _memory_span_size) && (insert_count < count)) { +#endif + span_t* current_span = span[insert_count++]; + current_span->next = cache->overflow; + cache->overflow = current_span; + } + atomic_store32_release(&cache->lock, 0); + + span_t* keep = 0; + for (size_t ispan = insert_count; ispan < count; ++ispan) { + span_t* current_span = span[ispan]; + // Keep master spans that has remaining subspans to avoid dangling them + if ((current_span->flags & SPAN_FLAG_MASTER) && + (atomic_load32(¤t_span->remaining_spans) > (int32_t)current_span->span_count)) { + current_span->next = keep; + keep = current_span; + } else { + _rpmalloc_span_unmap(current_span); + } + } + + if (keep) { + while (!atomic_cas32_acquire(&cache->lock, 1, 0)) + _rpmalloc_spin(); + + size_t islot = 0; + while (keep) { + for (; islot < cache->count; ++islot) { + span_t* current_span = cache->span[islot]; + if (!(current_span->flags & SPAN_FLAG_MASTER) || ((current_span->flags & SPAN_FLAG_MASTER) && + (atomic_load32(¤t_span->remaining_spans) <= (int32_t)current_span->span_count))) { + _rpmalloc_span_unmap(current_span); + cache->span[islot] = keep; + break; + } + } + if (islot == cache->count) + break; + keep = keep->next; + } + + if (keep) { + span_t* tail = keep; + while (tail->next) + tail = tail->next; + tail->next = cache->overflow; + cache->overflow = keep; + } + + atomic_store32_release(&cache->lock, 0); + } +} + +static size_t +_rpmalloc_global_cache_extract_spans(span_t** span, size_t span_count, size_t count) { + global_cache_t* cache = &_memory_span_cache[span_count - 1]; + + size_t extract_count = 0; + while (!atomic_cas32_acquire(&cache->lock, 1, 0)) + _rpmalloc_spin(); + +#if ENABLE_STATISTICS + cache->extract_count += count; +#endif + size_t want = count - extract_count; + if (want > cache->count) + want = cache->count; + + memcpy(span + extract_count, cache->span + (cache->count - want), sizeof(span_t*) * want); + cache->count -= (uint32_t)want; + extract_count += want; + + while ((extract_count < count) && cache->overflow) { + span_t* current_span = cache->overflow; + span[extract_count++] = current_span; + cache->overflow = current_span->next; + } + +#if ENABLE_ASSERTS + for (size_t ispan = 0; ispan < extract_count; ++ispan) { + assert(span[ispan]->span_count == span_count); + } +#endif + + atomic_store32_release(&cache->lock, 0); + + return extract_count; +} + +#endif + +//////////// +/// +/// Heap control +/// +////// + +static void _rpmalloc_deallocate_huge(span_t*); + +//! Store the given spans as reserve in the given heap +static void +_rpmalloc_heap_set_reserved_spans(heap_t* heap, span_t* master, span_t* reserve, size_t reserve_span_count) { + heap->span_reserve_master = master; + heap->span_reserve = reserve; + heap->spans_reserved = (uint32_t)reserve_span_count; +} + +//! Adopt the deferred span cache list, optionally extracting the first single span for immediate re-use +static void +_rpmalloc_heap_cache_adopt_deferred(heap_t* heap, span_t** single_span) { + span_t* span = (span_t*)((void*)atomic_exchange_ptr_acquire(&heap->span_free_deferred, 0)); + while (span) { + span_t* next_span = (span_t*)span->free_list; + rpmalloc_assert(span->heap == heap, "Span heap pointer corrupted"); + if (EXPECTED(span->size_class < SIZE_CLASS_COUNT)) { + rpmalloc_assert(heap->full_span_count, "Heap span counter corrupted"); + --heap->full_span_count; + _rpmalloc_stat_dec(&heap->span_use[0].spans_deferred); +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_remove(&heap->full_span[span->size_class], span); +#endif + _rpmalloc_stat_dec(&heap->span_use[0].current); + _rpmalloc_stat_dec(&heap->size_class_use[span->size_class].spans_current); + if (single_span && !*single_span) + *single_span = span; + else + _rpmalloc_heap_cache_insert(heap, span); + } else { + if (span->size_class == SIZE_CLASS_HUGE) { + _rpmalloc_deallocate_huge(span); + } else { + rpmalloc_assert(span->size_class == SIZE_CLASS_LARGE, "Span size class invalid"); + rpmalloc_assert(heap->full_span_count, "Heap span counter corrupted"); + --heap->full_span_count; +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_remove(&heap->large_huge_span, span); +#endif + uint32_t idx = span->span_count - 1; + _rpmalloc_stat_dec(&heap->span_use[idx].spans_deferred); + _rpmalloc_stat_dec(&heap->span_use[idx].current); + if (!idx && single_span && !*single_span) + *single_span = span; + else + _rpmalloc_heap_cache_insert(heap, span); + } + } + span = next_span; + } +} + +static void +_rpmalloc_heap_unmap(heap_t* heap) { + if (!heap->master_heap) { + if ((heap->finalize > 1) && !atomic_load32(&heap->child_count)) { + span_t* span = (span_t*)((uintptr_t)heap & _memory_span_mask); + _rpmalloc_span_unmap(span); + } + } else { + if (atomic_decr32(&heap->master_heap->child_count) == 0) { + _rpmalloc_heap_unmap(heap->master_heap); + } + } +} + +static void +_rpmalloc_heap_global_finalize(heap_t* heap) { + if (heap->finalize++ > 1) { + --heap->finalize; + return; + } + + _rpmalloc_heap_finalize(heap); + +#if ENABLE_THREAD_CACHE + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + span_cache_t* span_cache; + if (!iclass) + span_cache = &heap->span_cache; + else + span_cache = (span_cache_t*)(heap->span_large_cache + (iclass - 1)); + for (size_t ispan = 0; ispan < span_cache->count; ++ispan) + _rpmalloc_span_unmap(span_cache->span[ispan]); + span_cache->count = 0; + } +#endif + + if (heap->full_span_count) { + --heap->finalize; + return; + } + + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + if (heap->size_class[iclass].free_list || heap->size_class[iclass].partial_span) { + --heap->finalize; + return; + } + } + //Heap is now completely free, unmap and remove from heap list + size_t list_idx = (size_t)heap->id % HEAP_ARRAY_SIZE; + heap_t* list_heap = _memory_heaps[list_idx]; + if (list_heap == heap) { + _memory_heaps[list_idx] = heap->next_heap; + } else { + while (list_heap->next_heap != heap) + list_heap = list_heap->next_heap; + list_heap->next_heap = heap->next_heap; + } + + _rpmalloc_heap_unmap(heap); +} + +//! Insert a single span into thread heap cache, releasing to global cache if overflow +static void +_rpmalloc_heap_cache_insert(heap_t* heap, span_t* span) { + if (UNEXPECTED(heap->finalize != 0)) { + _rpmalloc_span_unmap(span); + _rpmalloc_heap_global_finalize(heap); + return; + } +#if ENABLE_THREAD_CACHE + size_t span_count = span->span_count; + _rpmalloc_stat_inc(&heap->span_use[span_count - 1].spans_to_cache); + if (span_count == 1) { + span_cache_t* span_cache = &heap->span_cache; + span_cache->span[span_cache->count++] = span; + if (span_cache->count == MAX_THREAD_SPAN_CACHE) { + const size_t remain_count = MAX_THREAD_SPAN_CACHE - THREAD_SPAN_CACHE_TRANSFER; +#if ENABLE_GLOBAL_CACHE + _rpmalloc_stat_add64(&heap->thread_to_global, THREAD_SPAN_CACHE_TRANSFER * _memory_span_size); + _rpmalloc_stat_add(&heap->span_use[span_count - 1].spans_to_global, THREAD_SPAN_CACHE_TRANSFER); + _rpmalloc_global_cache_insert_spans(span_cache->span + remain_count, span_count, THREAD_SPAN_CACHE_TRANSFER); +#else + for (size_t ispan = 0; ispan < THREAD_SPAN_CACHE_TRANSFER; ++ispan) + _rpmalloc_span_unmap(span_cache->span[remain_count + ispan]); +#endif + span_cache->count = remain_count; + } + } else { + size_t cache_idx = span_count - 2; + span_large_cache_t* span_cache = heap->span_large_cache + cache_idx; + span_cache->span[span_cache->count++] = span; + const size_t cache_limit = (MAX_THREAD_SPAN_LARGE_CACHE - (span_count >> 1)); + if (span_cache->count == cache_limit) { + const size_t transfer_limit = 2 + (cache_limit >> 2); + const size_t transfer_count = (THREAD_SPAN_LARGE_CACHE_TRANSFER <= transfer_limit ? THREAD_SPAN_LARGE_CACHE_TRANSFER : transfer_limit); + const size_t remain_count = cache_limit - transfer_count; +#if ENABLE_GLOBAL_CACHE + _rpmalloc_stat_add64(&heap->thread_to_global, transfer_count * span_count * _memory_span_size); + _rpmalloc_stat_add(&heap->span_use[span_count - 1].spans_to_global, transfer_count); + _rpmalloc_global_cache_insert_spans(span_cache->span + remain_count, span_count, transfer_count); +#else + for (size_t ispan = 0; ispan < transfer_count; ++ispan) + _rpmalloc_span_unmap(span_cache->span[remain_count + ispan]); +#endif + span_cache->count = remain_count; + } + } +#else + (void)sizeof(heap); + _rpmalloc_span_unmap(span); +#endif +} + +//! Extract the given number of spans from the different cache levels +static span_t* +_rpmalloc_heap_thread_cache_extract(heap_t* heap, size_t span_count) { + span_t* span = 0; +#if ENABLE_THREAD_CACHE + span_cache_t* span_cache; + if (span_count == 1) + span_cache = &heap->span_cache; + else + span_cache = (span_cache_t*)(heap->span_large_cache + (span_count - 2)); + if (span_cache->count) { + _rpmalloc_stat_inc(&heap->span_use[span_count - 1].spans_from_cache); + return span_cache->span[--span_cache->count]; + } +#endif + return span; +} + +static span_t* +_rpmalloc_heap_thread_cache_deferred_extract(heap_t* heap, size_t span_count) { + span_t* span = 0; + if (span_count == 1) { + _rpmalloc_heap_cache_adopt_deferred(heap, &span); + } else { + _rpmalloc_heap_cache_adopt_deferred(heap, 0); + span = _rpmalloc_heap_thread_cache_extract(heap, span_count); + } + return span; +} + +static span_t* +_rpmalloc_heap_reserved_extract(heap_t* heap, size_t span_count) { + if (heap->spans_reserved >= span_count) + return _rpmalloc_span_map(heap, span_count); + return 0; +} + +//! Extract a span from the global cache +static span_t* +_rpmalloc_heap_global_cache_extract(heap_t* heap, size_t span_count) { +#if ENABLE_GLOBAL_CACHE +#if ENABLE_THREAD_CACHE + span_cache_t* span_cache; + size_t wanted_count; + if (span_count == 1) { + span_cache = &heap->span_cache; + wanted_count = THREAD_SPAN_CACHE_TRANSFER; + } else { + span_cache = (span_cache_t*)(heap->span_large_cache + (span_count - 2)); + wanted_count = THREAD_SPAN_LARGE_CACHE_TRANSFER; + } + span_cache->count = _rpmalloc_global_cache_extract_spans(span_cache->span, span_count, wanted_count); + if (span_cache->count) { + _rpmalloc_stat_add64(&heap->global_to_thread, span_count * span_cache->count * _memory_span_size); + _rpmalloc_stat_add(&heap->span_use[span_count - 1].spans_from_global, span_cache->count); + return span_cache->span[--span_cache->count]; + } +#else + span_t* span = 0; + size_t count = _rpmalloc_global_cache_extract_spans(&span, span_count, 1); + if (count) { + _rpmalloc_stat_add64(&heap->global_to_thread, span_count * count * _memory_span_size); + _rpmalloc_stat_add(&heap->span_use[span_count - 1].spans_from_global, count); + return span; + } +#endif +#endif + (void)sizeof(heap); + (void)sizeof(span_count); + return 0; +} + +static void +_rpmalloc_inc_span_statistics(heap_t* heap, size_t span_count, uint32_t class_idx) { + (void)sizeof(heap); + (void)sizeof(span_count); + (void)sizeof(class_idx); +#if ENABLE_ADAPTIVE_THREAD_CACHE || ENABLE_STATISTICS + uint32_t idx = (uint32_t)span_count - 1; + uint32_t current_count = (uint32_t)atomic_incr32(&heap->span_use[idx].current); + if (current_count > (uint32_t)atomic_load32(&heap->span_use[idx].high)) + atomic_store32(&heap->span_use[idx].high, (int32_t)current_count); + _rpmalloc_stat_add_peak(&heap->size_class_use[class_idx].spans_current, 1, heap->size_class_use[class_idx].spans_peak); +#endif +} + +//! Get a span from one of the cache levels (thread cache, reserved, global cache) or fallback to mapping more memory +static span_t* +_rpmalloc_heap_extract_new_span(heap_t* heap, heap_size_class_t* heap_size_class, size_t span_count, uint32_t class_idx) { + span_t* span; +#if ENABLE_THREAD_CACHE + if (heap_size_class && heap_size_class->cache) { + span = heap_size_class->cache; + heap_size_class->cache = (heap->span_cache.count ? heap->span_cache.span[--heap->span_cache.count] : 0); + _rpmalloc_inc_span_statistics(heap, span_count, class_idx); + return span; + } +#endif + (void)sizeof(class_idx); + // Allow 50% overhead to increase cache hits + size_t base_span_count = span_count; + size_t limit_span_count = (span_count > 2) ? (span_count + (span_count >> 1)) : span_count; + if (limit_span_count > LARGE_CLASS_COUNT) + limit_span_count = LARGE_CLASS_COUNT; + do { + span = _rpmalloc_heap_thread_cache_extract(heap, span_count); + if (EXPECTED(span != 0)) { + _rpmalloc_stat_inc(&heap->size_class_use[class_idx].spans_from_cache); + _rpmalloc_inc_span_statistics(heap, span_count, class_idx); + return span; + } + span = _rpmalloc_heap_thread_cache_deferred_extract(heap, span_count); + if (EXPECTED(span != 0)) { + _rpmalloc_stat_inc(&heap->size_class_use[class_idx].spans_from_cache); + _rpmalloc_inc_span_statistics(heap, span_count, class_idx); + return span; + } + span = _rpmalloc_heap_reserved_extract(heap, span_count); + if (EXPECTED(span != 0)) { + _rpmalloc_stat_inc(&heap->size_class_use[class_idx].spans_from_reserved); + _rpmalloc_inc_span_statistics(heap, span_count, class_idx); + return span; + } + span = _rpmalloc_heap_global_cache_extract(heap, span_count); + if (EXPECTED(span != 0)) { + _rpmalloc_stat_inc(&heap->size_class_use[class_idx].spans_from_cache); + _rpmalloc_inc_span_statistics(heap, span_count, class_idx); + return span; + } + ++span_count; + } while (span_count <= limit_span_count); + //Final fallback, map in more virtual memory + span = _rpmalloc_span_map(heap, base_span_count); + _rpmalloc_inc_span_statistics(heap, base_span_count, class_idx); + _rpmalloc_stat_inc(&heap->size_class_use[class_idx].spans_map_calls); + return span; +} + +static void +_rpmalloc_heap_initialize(heap_t* heap) { + _rpmalloc_memset_const(heap, 0, sizeof(heap_t)); + //Get a new heap ID + heap->id = 1 + atomic_incr32(&_memory_heap_id); + + //Link in heap in heap ID map + size_t list_idx = (size_t)heap->id % HEAP_ARRAY_SIZE; + heap->next_heap = _memory_heaps[list_idx]; + _memory_heaps[list_idx] = heap; +} + +static void +_rpmalloc_heap_orphan(heap_t* heap, int first_class) { + heap->owner_thread = (uintptr_t)-1; +#if RPMALLOC_FIRST_CLASS_HEAPS + heap_t** heap_list = (first_class ? &_memory_first_class_orphan_heaps : &_memory_orphan_heaps); +#else + (void)sizeof(first_class); + heap_t** heap_list = &_memory_orphan_heaps; +#endif + heap->next_orphan = *heap_list; + *heap_list = heap; +} + +//! Allocate a new heap from newly mapped memory pages +static heap_t* +_rpmalloc_heap_allocate_new(void) { + // Map in pages for a 16 heaps. If page size is greater than required size for this, map a page and + // use first part for heaps and remaining part for spans for allocations. Adds a lot of complexity, + // but saves a lot of memory on systems where page size > 64 spans (4MiB) + size_t heap_size = sizeof(heap_t); + size_t aligned_heap_size = 16 * ((heap_size + 15) / 16); + size_t request_heap_count = 16; + size_t heap_span_count = ((aligned_heap_size * request_heap_count) + sizeof(span_t) + _memory_span_size - 1) / _memory_span_size; + size_t block_size = _memory_span_size * heap_span_count; + size_t span_count = heap_span_count; + span_t* span = 0; + // If there are global reserved spans, use these first + if (_memory_global_reserve_count >= heap_span_count) { + span = _rpmalloc_global_get_reserved_spans(heap_span_count); + } + if (!span) { + if (_memory_page_size > block_size) { + span_count = _memory_page_size / _memory_span_size; + block_size = _memory_page_size; + // If using huge pages, make sure to grab enough heaps to avoid reallocating a huge page just to serve new heaps + size_t possible_heap_count = (block_size - sizeof(span_t)) / aligned_heap_size; + if (possible_heap_count >= (request_heap_count * 16)) + request_heap_count *= 16; + else if (possible_heap_count < request_heap_count) + request_heap_count = possible_heap_count; + heap_span_count = ((aligned_heap_size * request_heap_count) + sizeof(span_t) + _memory_span_size - 1) / _memory_span_size; + } + + size_t align_offset = 0; + span = (span_t*)_rpmalloc_mmap(block_size, &align_offset); + if (!span) + return 0; + + // Master span will contain the heaps + _rpmalloc_stat_inc(&_master_spans); + _rpmalloc_span_initialize(span, span_count, heap_span_count, align_offset); + } + + size_t remain_size = _memory_span_size - sizeof(span_t); + heap_t* heap = (heap_t*)pointer_offset(span, sizeof(span_t)); + _rpmalloc_heap_initialize(heap); + + // Put extra heaps as orphans + size_t num_heaps = remain_size / aligned_heap_size; + if (num_heaps < request_heap_count) + num_heaps = request_heap_count; + atomic_store32(&heap->child_count, (int32_t)num_heaps - 1); + heap_t* extra_heap = (heap_t*)pointer_offset(heap, aligned_heap_size); + while (num_heaps > 1) { + _rpmalloc_heap_initialize(extra_heap); + extra_heap->master_heap = heap; + _rpmalloc_heap_orphan(extra_heap, 1); + extra_heap = (heap_t*)pointer_offset(extra_heap, aligned_heap_size); + --num_heaps; + } + + if (span_count > heap_span_count) { + // Cap reserved spans + size_t remain_count = span_count - heap_span_count; + size_t reserve_count = (remain_count > _memory_heap_reserve_count ? _memory_heap_reserve_count : remain_count); + span_t* remain_span = (span_t*)pointer_offset(span, heap_span_count * _memory_span_size); + _rpmalloc_heap_set_reserved_spans(heap, span, remain_span, reserve_count); + + if (remain_count > reserve_count) { + // Set to global reserved spans + remain_span = (span_t*)pointer_offset(remain_span, reserve_count * _memory_span_size); + reserve_count = remain_count - reserve_count; + _rpmalloc_global_set_reserved_spans(span, remain_span, reserve_count); + } + } + + return heap; +} + +static heap_t* +_rpmalloc_heap_extract_orphan(heap_t** heap_list) { + heap_t* heap = *heap_list; + *heap_list = (heap ? heap->next_orphan : 0); + return heap; +} + +//! Allocate a new heap, potentially reusing a previously orphaned heap +static heap_t* +_rpmalloc_heap_allocate(int first_class) { + heap_t* heap = 0; + while (!atomic_cas32_acquire(&_memory_global_lock, 1, 0)) + _rpmalloc_spin(); + if (first_class == 0) + heap = _rpmalloc_heap_extract_orphan(&_memory_orphan_heaps); +#if RPMALLOC_FIRST_CLASS_HEAPS + if (!heap) + heap = _rpmalloc_heap_extract_orphan(&_memory_first_class_orphan_heaps); +#endif + if (!heap) + heap = _rpmalloc_heap_allocate_new(); + atomic_store32_release(&_memory_global_lock, 0); + _rpmalloc_heap_cache_adopt_deferred(heap, 0); + return heap; +} + +static void +_rpmalloc_heap_release(void* heapptr, int first_class, int release_cache) { + heap_t* heap = (heap_t*)heapptr; + if (!heap) + return; + //Release thread cache spans back to global cache + _rpmalloc_heap_cache_adopt_deferred(heap, 0); + if (release_cache || heap->finalize) { +#if ENABLE_THREAD_CACHE + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + span_cache_t* span_cache; + if (!iclass) + span_cache = &heap->span_cache; + else + span_cache = (span_cache_t*)(heap->span_large_cache + (iclass - 1)); + if (!span_cache->count) + continue; +#if ENABLE_GLOBAL_CACHE + if (heap->finalize) { + for (size_t ispan = 0; ispan < span_cache->count; ++ispan) + _rpmalloc_span_unmap(span_cache->span[ispan]); + } else { + _rpmalloc_stat_add64(&heap->thread_to_global, span_cache->count * (iclass + 1) * _memory_span_size); + _rpmalloc_stat_add(&heap->span_use[iclass].spans_to_global, span_cache->count); + _rpmalloc_global_cache_insert_spans(span_cache->span, iclass + 1, span_cache->count); + } +#else + for (size_t ispan = 0; ispan < span_cache->count; ++ispan) + _rpmalloc_span_unmap(span_cache->span[ispan]); +#endif + span_cache->count = 0; + } +#endif + } + + if (get_thread_heap_raw() == heap) + set_thread_heap(0); + +#if ENABLE_STATISTICS + atomic_decr32(&_memory_active_heaps); + rpmalloc_assert(atomic_load32(&_memory_active_heaps) >= 0, "Still active heaps during finalization"); +#endif + + // If we are forcibly terminating with _exit the state of the + // lock atomic is unknown and it's best to just go ahead and exit + if (get_thread_id() != _rpmalloc_main_thread_id) { + while (!atomic_cas32_acquire(&_memory_global_lock, 1, 0)) + _rpmalloc_spin(); + } + _rpmalloc_heap_orphan(heap, first_class); + atomic_store32_release(&_memory_global_lock, 0); +} + +static void +_rpmalloc_heap_release_raw(void* heapptr, int release_cache) { + _rpmalloc_heap_release(heapptr, 0, release_cache); +} + +static void +_rpmalloc_heap_release_raw_fc(void* heapptr) { + _rpmalloc_heap_release_raw(heapptr, 1); +} + +static void +_rpmalloc_heap_finalize(heap_t* heap) { + if (heap->spans_reserved) { + span_t* span = _rpmalloc_span_map(heap, heap->spans_reserved); + _rpmalloc_span_unmap(span); + heap->spans_reserved = 0; + } + + _rpmalloc_heap_cache_adopt_deferred(heap, 0); + + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + if (heap->size_class[iclass].cache) + _rpmalloc_span_unmap(heap->size_class[iclass].cache); + heap->size_class[iclass].cache = 0; + span_t* span = heap->size_class[iclass].partial_span; + while (span) { + span_t* next = span->next; + _rpmalloc_span_finalize(heap, iclass, span, &heap->size_class[iclass].partial_span); + span = next; + } + // If class still has a free list it must be a full span + if (heap->size_class[iclass].free_list) { + span_t* class_span = (span_t*)((uintptr_t)heap->size_class[iclass].free_list & _memory_span_mask); + span_t** list = 0; +#if RPMALLOC_FIRST_CLASS_HEAPS + list = &heap->full_span[iclass]; +#endif + --heap->full_span_count; + if (!_rpmalloc_span_finalize(heap, iclass, class_span, list)) { + if (list) + _rpmalloc_span_double_link_list_remove(list, class_span); + _rpmalloc_span_double_link_list_add(&heap->size_class[iclass].partial_span, class_span); + } + } + } + +#if ENABLE_THREAD_CACHE + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + span_cache_t* span_cache; + if (!iclass) + span_cache = &heap->span_cache; + else + span_cache = (span_cache_t*)(heap->span_large_cache + (iclass - 1)); + for (size_t ispan = 0; ispan < span_cache->count; ++ispan) + _rpmalloc_span_unmap(span_cache->span[ispan]); + span_cache->count = 0; + } +#endif + rpmalloc_assert(!atomic_load_ptr(&heap->span_free_deferred), "Heaps still active during finalization"); +} + + +//////////// +/// +/// Allocation entry points +/// +////// + +//! Pop first block from a free list +static void* +free_list_pop(void** list) { + void* block = *list; + *list = *((void**)block); + return block; +} + +//! Allocate a small/medium sized memory block from the given heap +static void* +_rpmalloc_allocate_from_heap_fallback(heap_t* heap, heap_size_class_t* heap_size_class, uint32_t class_idx) { + span_t* span = heap_size_class->partial_span; + if (EXPECTED(span != 0)) { + rpmalloc_assert(span->block_count == _memory_size_class[span->size_class].block_count, "Span block count corrupted"); + rpmalloc_assert(!_rpmalloc_span_is_fully_utilized(span), "Internal failure"); + void* block; + if (span->free_list) { + //Span local free list is not empty, swap to size class free list + block = free_list_pop(&span->free_list); + heap_size_class->free_list = span->free_list; + span->free_list = 0; + } else { + //If the span did not fully initialize free list, link up another page worth of blocks + void* block_start = pointer_offset(span, SPAN_HEADER_SIZE + ((size_t)span->free_list_limit * span->block_size)); + span->free_list_limit += free_list_partial_init(&heap_size_class->free_list, &block, + (void*)((uintptr_t)block_start & ~(_memory_page_size - 1)), block_start, + span->block_count - span->free_list_limit, span->block_size); + } + rpmalloc_assert(span->free_list_limit <= span->block_count, "Span block count corrupted"); + span->used_count = span->free_list_limit; + + //Swap in deferred free list if present + if (atomic_load_ptr(&span->free_list_deferred)) + _rpmalloc_span_extract_free_list_deferred(span); + + //If span is still not fully utilized keep it in partial list and early return block + if (!_rpmalloc_span_is_fully_utilized(span)) + return block; + + //The span is fully utilized, unlink from partial list and add to fully utilized list + _rpmalloc_span_double_link_list_pop_head(&heap_size_class->partial_span, span); +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_add(&heap->full_span[class_idx], span); +#endif + ++heap->full_span_count; + return block; + } + + //Find a span in one of the cache levels + span = _rpmalloc_heap_extract_new_span(heap, heap_size_class, 1, class_idx); + if (EXPECTED(span != 0)) { + //Mark span as owned by this heap and set base data, return first block + return _rpmalloc_span_initialize_new(heap, heap_size_class, span, class_idx); + } + + return 0; +} + +//! Allocate a small sized memory block from the given heap +static void* +_rpmalloc_allocate_small(heap_t* heap, size_t size) { + rpmalloc_assert(heap, "No thread heap"); + //Small sizes have unique size classes + const uint32_t class_idx = (uint32_t)((size + (SMALL_GRANULARITY - 1)) >> SMALL_GRANULARITY_SHIFT); + heap_size_class_t* heap_size_class = heap->size_class + class_idx; + _rpmalloc_stat_inc_alloc(heap, class_idx); + if (EXPECTED(heap_size_class->free_list != 0)) + return free_list_pop(&heap_size_class->free_list); + return _rpmalloc_allocate_from_heap_fallback(heap, heap_size_class, class_idx); +} + +//! Allocate a medium sized memory block from the given heap +static void* +_rpmalloc_allocate_medium(heap_t* heap, size_t size) { + rpmalloc_assert(heap, "No thread heap"); + //Calculate the size class index and do a dependent lookup of the final class index (in case of merged classes) + const uint32_t base_idx = (uint32_t)(SMALL_CLASS_COUNT + ((size - (SMALL_SIZE_LIMIT + 1)) >> MEDIUM_GRANULARITY_SHIFT)); + const uint32_t class_idx = _memory_size_class[base_idx].class_idx; + heap_size_class_t* heap_size_class = heap->size_class + class_idx; + _rpmalloc_stat_inc_alloc(heap, class_idx); + if (EXPECTED(heap_size_class->free_list != 0)) + return free_list_pop(&heap_size_class->free_list); + return _rpmalloc_allocate_from_heap_fallback(heap, heap_size_class, class_idx); +} + +//! Allocate a large sized memory block from the given heap +static void* +_rpmalloc_allocate_large(heap_t* heap, size_t size) { + rpmalloc_assert(heap, "No thread heap"); + //Calculate number of needed max sized spans (including header) + //Since this function is never called if size > LARGE_SIZE_LIMIT + //the span_count is guaranteed to be <= LARGE_CLASS_COUNT + size += SPAN_HEADER_SIZE; + size_t span_count = size >> _memory_span_size_shift; + if (size & (_memory_span_size - 1)) + ++span_count; + + //Find a span in one of the cache levels + span_t* span = _rpmalloc_heap_extract_new_span(heap, 0, span_count, SIZE_CLASS_LARGE); + if (!span) + return span; + + //Mark span as owned by this heap and set base data + rpmalloc_assert(span->span_count >= span_count, "Internal failure"); + span->size_class = SIZE_CLASS_LARGE; + span->heap = heap; + +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_add(&heap->large_huge_span, span); +#endif + ++heap->full_span_count; + + return pointer_offset(span, SPAN_HEADER_SIZE); +} + +//! Allocate a huge block by mapping memory pages directly +static void* +_rpmalloc_allocate_huge(heap_t* heap, size_t size) { + rpmalloc_assert(heap, "No thread heap"); + _rpmalloc_heap_cache_adopt_deferred(heap, 0); + size += SPAN_HEADER_SIZE; + size_t num_pages = size >> _memory_page_size_shift; + if (size & (_memory_page_size - 1)) + ++num_pages; + size_t align_offset = 0; + span_t* span = (span_t*)_rpmalloc_mmap(num_pages * _memory_page_size, &align_offset); + if (!span) + return span; + + //Store page count in span_count + span->size_class = SIZE_CLASS_HUGE; + span->span_count = (uint32_t)num_pages; + span->align_offset = (uint32_t)align_offset; + span->heap = heap; + _rpmalloc_stat_add_peak(&_huge_pages_current, num_pages, _huge_pages_peak); + +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_add(&heap->large_huge_span, span); +#endif + ++heap->full_span_count; + + return pointer_offset(span, SPAN_HEADER_SIZE); +} + +//! Allocate a block of the given size +static void* +_rpmalloc_allocate(heap_t* heap, size_t size) { + _rpmalloc_stat_add64(&_allocation_counter, 1); + if (EXPECTED(size <= SMALL_SIZE_LIMIT)) + return _rpmalloc_allocate_small(heap, size); + else if (size <= _memory_medium_size_limit) + return _rpmalloc_allocate_medium(heap, size); + else if (size <= LARGE_SIZE_LIMIT) + return _rpmalloc_allocate_large(heap, size); + return _rpmalloc_allocate_huge(heap, size); +} + +static void* +_rpmalloc_aligned_allocate(heap_t* heap, size_t alignment, size_t size) { + if (alignment <= SMALL_GRANULARITY) + return _rpmalloc_allocate(heap, size); + +#if ENABLE_VALIDATE_ARGS + if ((size + alignment) < size) { + errno = EINVAL; + return 0; + } + if (alignment & (alignment - 1)) { + errno = EINVAL; + return 0; + } +#endif + + if ((alignment <= SPAN_HEADER_SIZE) && (size < _memory_medium_size_limit)) { + // If alignment is less or equal to span header size (which is power of two), + // and size aligned to span header size multiples is less than size + alignment, + // then use natural alignment of blocks to provide alignment + size_t multiple_size = size ? (size + (SPAN_HEADER_SIZE - 1)) & ~(uintptr_t)(SPAN_HEADER_SIZE - 1) : SPAN_HEADER_SIZE; + rpmalloc_assert(!(multiple_size % SPAN_HEADER_SIZE), "Failed alignment calculation"); + if (multiple_size <= (size + alignment)) + return _rpmalloc_allocate(heap, multiple_size); + } + + void* ptr = 0; + size_t align_mask = alignment - 1; + if (alignment <= _memory_page_size) { + ptr = _rpmalloc_allocate(heap, size + alignment); + if ((uintptr_t)ptr & align_mask) { + ptr = (void*)(((uintptr_t)ptr & ~(uintptr_t)align_mask) + alignment); + //Mark as having aligned blocks + span_t* span = (span_t*)((uintptr_t)ptr & _memory_span_mask); + span->flags |= SPAN_FLAG_ALIGNED_BLOCKS; + } + return ptr; + } + + // Fallback to mapping new pages for this request. Since pointers passed + // to rpfree must be able to reach the start of the span by bitmasking of + // the address with the span size, the returned aligned pointer from this + // function must be with a span size of the start of the mapped area. + // In worst case this requires us to loop and map pages until we get a + // suitable memory address. It also means we can never align to span size + // or greater, since the span header will push alignment more than one + // span size away from span start (thus causing pointer mask to give us + // an invalid span start on free) + if (alignment & align_mask) { + errno = EINVAL; + return 0; + } + if (alignment >= _memory_span_size) { + errno = EINVAL; + return 0; + } + + size_t extra_pages = alignment / _memory_page_size; + + // Since each span has a header, we will at least need one extra memory page + size_t num_pages = 1 + (size / _memory_page_size); + if (size & (_memory_page_size - 1)) + ++num_pages; + + if (extra_pages > num_pages) + num_pages = 1 + extra_pages; + + size_t original_pages = num_pages; + size_t limit_pages = (_memory_span_size / _memory_page_size) * 2; + if (limit_pages < (original_pages * 2)) + limit_pages = original_pages * 2; + + size_t mapped_size, align_offset; + span_t* span; + +retry: + align_offset = 0; + mapped_size = num_pages * _memory_page_size; + + span = (span_t*)_rpmalloc_mmap(mapped_size, &align_offset); + if (!span) { + errno = ENOMEM; + return 0; + } + ptr = pointer_offset(span, SPAN_HEADER_SIZE); + + if ((uintptr_t)ptr & align_mask) + ptr = (void*)(((uintptr_t)ptr & ~(uintptr_t)align_mask) + alignment); + + if (((size_t)pointer_diff(ptr, span) >= _memory_span_size) || + (pointer_offset(ptr, size) > pointer_offset(span, mapped_size)) || + (((uintptr_t)ptr & _memory_span_mask) != (uintptr_t)span)) { + _rpmalloc_unmap(span, mapped_size, align_offset, mapped_size); + ++num_pages; + if (num_pages > limit_pages) { + errno = EINVAL; + return 0; + } + goto retry; + } + + //Store page count in span_count + span->size_class = SIZE_CLASS_HUGE; + span->span_count = (uint32_t)num_pages; + span->align_offset = (uint32_t)align_offset; + span->heap = heap; + _rpmalloc_stat_add_peak(&_huge_pages_current, num_pages, _huge_pages_peak); + +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_add(&heap->large_huge_span, span); +#endif + ++heap->full_span_count; + + _rpmalloc_stat_add64(&_allocation_counter, 1); + + return ptr; +} + + +//////////// +/// +/// Deallocation entry points +/// +////// + +//! Deallocate the given small/medium memory block in the current thread local heap +static void +_rpmalloc_deallocate_direct_small_or_medium(span_t* span, void* block) { + heap_t* heap = span->heap; + rpmalloc_assert(heap->owner_thread == get_thread_id() || !heap->owner_thread || heap->finalize, "Internal failure"); + //Add block to free list + if (UNEXPECTED(_rpmalloc_span_is_fully_utilized(span))) { + span->used_count = span->block_count; +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_remove(&heap->full_span[span->size_class], span); +#endif + _rpmalloc_span_double_link_list_add(&heap->size_class[span->size_class].partial_span, span); + --heap->full_span_count; + } + *((void**)block) = span->free_list; + --span->used_count; + span->free_list = block; + if (UNEXPECTED(span->used_count == span->list_size)) { + // If there are no used blocks it is guaranteed that no other external thread is accessing the span + if (span->used_count) { + // Make sure we have synchronized the deferred list and list size by using acquire semantics + // and guarantee that no external thread is accessing span concurrently + void* free_list; + do { + free_list = atomic_exchange_ptr_acquire(&span->free_list_deferred, INVALID_POINTER); + } while (free_list == INVALID_POINTER); + atomic_store_ptr_release(&span->free_list_deferred, free_list); + } + _rpmalloc_span_double_link_list_remove(&heap->size_class[span->size_class].partial_span, span); + _rpmalloc_span_release_to_cache(heap, span); + } +} + +static void +_rpmalloc_deallocate_defer_free_span(heap_t* heap, span_t* span) { + if (span->size_class != SIZE_CLASS_HUGE) + _rpmalloc_stat_inc(&heap->span_use[span->span_count - 1].spans_deferred); + //This list does not need ABA protection, no mutable side state + do { + span->free_list = (void*)atomic_load_ptr(&heap->span_free_deferred); + } while (!atomic_cas_ptr(&heap->span_free_deferred, span, span->free_list)); +} + +//! Put the block in the deferred free list of the owning span +static void +_rpmalloc_deallocate_defer_small_or_medium(span_t* span, void* block) { + // The memory ordering here is a bit tricky, to avoid having to ABA protect + // the deferred free list to avoid desynchronization of list and list size + // we need to have acquire semantics on successful CAS of the pointer to + // guarantee the list_size variable validity + release semantics on pointer store + void* free_list; + do { + free_list = atomic_exchange_ptr_acquire(&span->free_list_deferred, INVALID_POINTER); + } while (free_list == INVALID_POINTER); + *((void**)block) = free_list; + uint32_t free_count = ++span->list_size; + int all_deferred_free = (free_count == span->block_count); + atomic_store_ptr_release(&span->free_list_deferred, block); + if (all_deferred_free) { + // Span was completely freed by this block. Due to the INVALID_POINTER spin lock + // no other thread can reach this state simultaneously on this span. + // Safe to move to owner heap deferred cache + _rpmalloc_deallocate_defer_free_span(span->heap, span); + } +} + +static void +_rpmalloc_deallocate_small_or_medium(span_t* span, void* p) { + _rpmalloc_stat_inc_free(span->heap, span->size_class); + if (span->flags & SPAN_FLAG_ALIGNED_BLOCKS) { + //Realign pointer to block start + void* blocks_start = pointer_offset(span, SPAN_HEADER_SIZE); + uint32_t block_offset = (uint32_t)pointer_diff(p, blocks_start); + p = pointer_offset(p, -(int32_t)(block_offset % span->block_size)); + } + //Check if block belongs to this heap or if deallocation should be deferred +#if RPMALLOC_FIRST_CLASS_HEAPS + int defer = (span->heap->owner_thread && (span->heap->owner_thread != get_thread_id()) && !span->heap->finalize); +#else + int defer = ((span->heap->owner_thread != get_thread_id()) && !span->heap->finalize); +#endif + if (!defer) + _rpmalloc_deallocate_direct_small_or_medium(span, p); + else + _rpmalloc_deallocate_defer_small_or_medium(span, p); +} + +//! Deallocate the given large memory block to the current heap +static void +_rpmalloc_deallocate_large(span_t* span) { + rpmalloc_assert(span->size_class == SIZE_CLASS_LARGE, "Bad span size class"); + rpmalloc_assert(!(span->flags & SPAN_FLAG_MASTER) || !(span->flags & SPAN_FLAG_SUBSPAN), "Span flag corrupted"); + rpmalloc_assert((span->flags & SPAN_FLAG_MASTER) || (span->flags & SPAN_FLAG_SUBSPAN), "Span flag corrupted"); + //We must always defer (unless finalizing) if from another heap since we cannot touch the list or counters of another heap +#if RPMALLOC_FIRST_CLASS_HEAPS + int defer = (span->heap->owner_thread && (span->heap->owner_thread != get_thread_id()) && !span->heap->finalize); +#else + int defer = ((span->heap->owner_thread != get_thread_id()) && !span->heap->finalize); +#endif + if (defer) { + _rpmalloc_deallocate_defer_free_span(span->heap, span); + return; + } + rpmalloc_assert(span->heap->full_span_count, "Heap span counter corrupted"); + --span->heap->full_span_count; +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_remove(&span->heap->large_huge_span, span); +#endif +#if ENABLE_ADAPTIVE_THREAD_CACHE || ENABLE_STATISTICS + //Decrease counter + size_t idx = span->span_count - 1; + atomic_decr32(&span->heap->span_use[idx].current); +#endif + heap_t* heap = span->heap; + rpmalloc_assert(heap, "No thread heap"); +#if ENABLE_THREAD_CACHE + const int set_as_reserved = ((span->span_count > 1) && (heap->span_cache.count == 0) && !heap->finalize && !heap->spans_reserved); +#else + const int set_as_reserved = ((span->span_count > 1) && !heap->finalize && !heap->spans_reserved); +#endif + if (set_as_reserved) { + heap->span_reserve = span; + heap->spans_reserved = span->span_count; + if (span->flags & SPAN_FLAG_MASTER) { + heap->span_reserve_master = span; + } else { //SPAN_FLAG_SUBSPAN + span_t* master = (span_t*)pointer_offset(span, -(intptr_t)((size_t)span->offset_from_master * _memory_span_size)); + heap->span_reserve_master = master; + rpmalloc_assert(master->flags & SPAN_FLAG_MASTER, "Span flag corrupted"); + rpmalloc_assert(atomic_load32(&master->remaining_spans) >= (int32_t)span->span_count, "Master span count corrupted"); + } + _rpmalloc_stat_inc(&heap->span_use[idx].spans_to_reserved); + } else { + //Insert into cache list + _rpmalloc_heap_cache_insert(heap, span); + } +} + +//! Deallocate the given huge span +static void +_rpmalloc_deallocate_huge(span_t* span) { + rpmalloc_assert(span->heap, "No span heap"); +#if RPMALLOC_FIRST_CLASS_HEAPS + int defer = (span->heap->owner_thread && (span->heap->owner_thread != get_thread_id()) && !span->heap->finalize); +#else + int defer = ((span->heap->owner_thread != get_thread_id()) && !span->heap->finalize); +#endif + if (defer) { + _rpmalloc_deallocate_defer_free_span(span->heap, span); + return; + } + rpmalloc_assert(span->heap->full_span_count, "Heap span counter corrupted"); + --span->heap->full_span_count; +#if RPMALLOC_FIRST_CLASS_HEAPS + _rpmalloc_span_double_link_list_remove(&span->heap->large_huge_span, span); +#endif + + //Oversized allocation, page count is stored in span_count + size_t num_pages = span->span_count; + _rpmalloc_unmap(span, num_pages * _memory_page_size, span->align_offset, num_pages * _memory_page_size); + _rpmalloc_stat_sub(&_huge_pages_current, num_pages); +} + +//! Deallocate the given block +static void +_rpmalloc_deallocate(void* p) { + _rpmalloc_stat_add64(&_deallocation_counter, 1); + //Grab the span (always at start of span, using span alignment) + span_t* span = (span_t*)((uintptr_t)p & _memory_span_mask); + if (UNEXPECTED(!span)) + return; + if (EXPECTED(span->size_class < SIZE_CLASS_COUNT)) + _rpmalloc_deallocate_small_or_medium(span, p); + else if (span->size_class == SIZE_CLASS_LARGE) + _rpmalloc_deallocate_large(span); + else + _rpmalloc_deallocate_huge(span); +} + +//////////// +/// +/// Reallocation entry points +/// +////// + +static size_t +_rpmalloc_usable_size(void* p); + +//! Reallocate the given block to the given size +static void* +_rpmalloc_reallocate(heap_t* heap, void* p, size_t size, size_t oldsize, unsigned int flags) { + if (p) { + //Grab the span using guaranteed span alignment + span_t* span = (span_t*)((uintptr_t)p & _memory_span_mask); + if (EXPECTED(span->size_class < SIZE_CLASS_COUNT)) { + //Small/medium sized block + rpmalloc_assert(span->span_count == 1, "Span counter corrupted"); + void* blocks_start = pointer_offset(span, SPAN_HEADER_SIZE); + uint32_t block_offset = (uint32_t)pointer_diff(p, blocks_start); + uint32_t block_idx = block_offset / span->block_size; + void* block = pointer_offset(blocks_start, (size_t)block_idx * span->block_size); + if (!oldsize) + oldsize = (size_t)((ptrdiff_t)span->block_size - pointer_diff(p, block)); + if ((size_t)span->block_size >= size) { + //Still fits in block, never mind trying to save memory, but preserve data if alignment changed + if ((p != block) && !(flags & RPMALLOC_NO_PRESERVE)) + memmove(block, p, oldsize); + return block; + } + } else if (span->size_class == SIZE_CLASS_LARGE) { + //Large block + size_t total_size = size + SPAN_HEADER_SIZE; + size_t num_spans = total_size >> _memory_span_size_shift; + if (total_size & (_memory_span_mask - 1)) + ++num_spans; + size_t current_spans = span->span_count; + void* block = pointer_offset(span, SPAN_HEADER_SIZE); + if (!oldsize) + oldsize = (current_spans * _memory_span_size) - (size_t)pointer_diff(p, block) - SPAN_HEADER_SIZE; + if ((current_spans >= num_spans) && (total_size >= (oldsize / 2))) { + //Still fits in block, never mind trying to save memory, but preserve data if alignment changed + if ((p != block) && !(flags & RPMALLOC_NO_PRESERVE)) + memmove(block, p, oldsize); + return block; + } + } else { + //Oversized block + size_t total_size = size + SPAN_HEADER_SIZE; + size_t num_pages = total_size >> _memory_page_size_shift; + if (total_size & (_memory_page_size - 1)) + ++num_pages; + //Page count is stored in span_count + size_t current_pages = span->span_count; + void* block = pointer_offset(span, SPAN_HEADER_SIZE); + if (!oldsize) + oldsize = (current_pages * _memory_page_size) - (size_t)pointer_diff(p, block) - SPAN_HEADER_SIZE; + if ((current_pages >= num_pages) && (num_pages >= (current_pages / 2))) { + //Still fits in block, never mind trying to save memory, but preserve data if alignment changed + if ((p != block) && !(flags & RPMALLOC_NO_PRESERVE)) + memmove(block, p, oldsize); + return block; + } + } + } else { + oldsize = 0; + } + + if (!!(flags & RPMALLOC_GROW_OR_FAIL)) + return 0; + + //Size is greater than block size, need to allocate a new block and deallocate the old + //Avoid hysteresis by overallocating if increase is small (below 37%) + size_t lower_bound = oldsize + (oldsize >> 2) + (oldsize >> 3); + size_t new_size = (size > lower_bound) ? size : ((size > oldsize) ? lower_bound : size); + void* block = _rpmalloc_allocate(heap, new_size); + if (p && block) { + if (!(flags & RPMALLOC_NO_PRESERVE)) + memcpy(block, p, oldsize < new_size ? oldsize : new_size); + _rpmalloc_deallocate(p); + } + + return block; +} + +static void* +_rpmalloc_aligned_reallocate(heap_t* heap, void* ptr, size_t alignment, size_t size, size_t oldsize, + unsigned int flags) { + if (alignment <= SMALL_GRANULARITY) + return _rpmalloc_reallocate(heap, ptr, size, oldsize, flags); + + int no_alloc = !!(flags & RPMALLOC_GROW_OR_FAIL); + size_t usablesize = (ptr ? _rpmalloc_usable_size(ptr) : 0); + if ((usablesize >= size) && !((uintptr_t)ptr & (alignment - 1))) { + if (no_alloc || (size >= (usablesize / 2))) + return ptr; + } + // Aligned alloc marks span as having aligned blocks + void* block = (!no_alloc ? _rpmalloc_aligned_allocate(heap, alignment, size) : 0); + if (EXPECTED(block != 0)) { + if (!(flags & RPMALLOC_NO_PRESERVE) && ptr) { + if (!oldsize) + oldsize = usablesize; + memcpy(block, ptr, oldsize < size ? oldsize : size); + } + _rpmalloc_deallocate(ptr); + } + return block; +} + + +//////////// +/// +/// Initialization, finalization and utility +/// +////// + +//! Get the usable size of the given block +static size_t +_rpmalloc_usable_size(void* p) { + //Grab the span using guaranteed span alignment + span_t* span = (span_t*)((uintptr_t)p & _memory_span_mask); + if (span->size_class < SIZE_CLASS_COUNT) { + //Small/medium block + void* blocks_start = pointer_offset(span, SPAN_HEADER_SIZE); + return span->block_size - ((size_t)pointer_diff(p, blocks_start) % span->block_size); + } + if (span->size_class == SIZE_CLASS_LARGE) { + //Large block + size_t current_spans = span->span_count; + return (current_spans * _memory_span_size) - (size_t)pointer_diff(p, span); + } + //Oversized block, page count is stored in span_count + size_t current_pages = span->span_count; + return (current_pages * _memory_page_size) - (size_t)pointer_diff(p, span); +} + +//! Adjust and optimize the size class properties for the given class +static void +_rpmalloc_adjust_size_class(size_t iclass) { + size_t block_size = _memory_size_class[iclass].block_size; + size_t block_count = (_memory_span_size - SPAN_HEADER_SIZE) / block_size; + + _memory_size_class[iclass].block_count = (uint16_t)block_count; + _memory_size_class[iclass].class_idx = (uint16_t)iclass; + + //Check if previous size classes can be merged + if (iclass >= SMALL_CLASS_COUNT) { + size_t prevclass = iclass; + while (prevclass > 0) { + --prevclass; + //A class can be merged if number of pages and number of blocks are equal + if (_memory_size_class[prevclass].block_count == _memory_size_class[iclass].block_count) + _rpmalloc_memcpy_const(_memory_size_class + prevclass, _memory_size_class + iclass, sizeof(_memory_size_class[iclass])); + else + break; + } + } +} + +//! Initialize the allocator and setup global data +extern inline int +rpmalloc_initialize(void) { + if (_rpmalloc_initialized) { + rpmalloc_thread_initialize(); + return 0; + } + return rpmalloc_initialize_config(0); +} + +int +rpmalloc_initialize_config(const rpmalloc_config_t* config) { + if (_rpmalloc_initialized) { + rpmalloc_thread_initialize(); + return 0; + } + _rpmalloc_initialized = 1; + + if (config) + memcpy(&_memory_config, config, sizeof(rpmalloc_config_t)); + else + _rpmalloc_memset_const(&_memory_config, 0, sizeof(rpmalloc_config_t)); + + if (!_memory_config.memory_map || !_memory_config.memory_unmap) { + _memory_config.memory_map = _rpmalloc_mmap_os; + _memory_config.memory_unmap = _rpmalloc_unmap_os; + } + +#if PLATFORM_WINDOWS + SYSTEM_INFO system_info; + memset(&system_info, 0, sizeof(system_info)); + GetSystemInfo(&system_info); + _memory_map_granularity = system_info.dwAllocationGranularity; +#else + _memory_map_granularity = (size_t)sysconf(_SC_PAGESIZE); +#endif + +#if RPMALLOC_CONFIGURABLE + _memory_page_size = _memory_config.page_size; +#else + _memory_page_size = 0; +#endif + _memory_huge_pages = 0; + if (!_memory_page_size) { +#if PLATFORM_WINDOWS + _memory_page_size = system_info.dwPageSize; +#else + _memory_page_size = _memory_map_granularity; + if (_memory_config.enable_huge_pages) { +#if defined(__linux__) + size_t huge_page_size = 0; + FILE* meminfo = fopen("/proc/meminfo", "r"); + if (meminfo) { + char line[128]; + while (!huge_page_size && fgets(line, sizeof(line) - 1, meminfo)) { + line[sizeof(line) - 1] = 0; + if (strstr(line, "Hugepagesize:")) + huge_page_size = (size_t)strtol(line + 13, 0, 10) * 1024; + } + fclose(meminfo); + } + if (huge_page_size) { + _memory_huge_pages = 1; + _memory_page_size = huge_page_size; + _memory_map_granularity = huge_page_size; + } +#elif defined(__FreeBSD__) + int rc; + size_t sz = sizeof(rc); + + if (sysctlbyname("vm.pmap.pg_ps_enabled", &rc, &sz, NULL, 0) == 0 && rc == 1) { + _memory_huge_pages = 1; + _memory_page_size = 2 * 1024 * 1024; + _memory_map_granularity = _memory_page_size; + } +#elif defined(__APPLE__) || defined(__NetBSD__) + _memory_huge_pages = 1; + _memory_page_size = 2 * 1024 * 1024; + _memory_map_granularity = _memory_page_size; +#endif + } +#endif + } else { + if (_memory_config.enable_huge_pages) + _memory_huge_pages = 1; + } + +#if PLATFORM_WINDOWS + if (_memory_config.enable_huge_pages) { + HANDLE token = 0; + size_t large_page_minimum = GetLargePageMinimum(); + if (large_page_minimum) + OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &token); + if (token) { + LUID luid; + if (LookupPrivilegeValue(0, SE_LOCK_MEMORY_NAME, &luid)) { + TOKEN_PRIVILEGES token_privileges; + memset(&token_privileges, 0, sizeof(token_privileges)); + token_privileges.PrivilegeCount = 1; + token_privileges.Privileges[0].Luid = luid; + token_privileges.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED; + if (AdjustTokenPrivileges(token, FALSE, &token_privileges, 0, 0, 0)) { + if (GetLastError() == ERROR_SUCCESS) + _memory_huge_pages = 1; + } + } + CloseHandle(token); + } + if (_memory_huge_pages) { + if (large_page_minimum > _memory_page_size) + _memory_page_size = large_page_minimum; + if (large_page_minimum > _memory_map_granularity) + _memory_map_granularity = large_page_minimum; + } + } +#endif + + size_t min_span_size = 256; + size_t max_page_size; +#if UINTPTR_MAX > 0xFFFFFFFF + max_page_size = 4096ULL * 1024ULL * 1024ULL; +#else + max_page_size = 4 * 1024 * 1024; +#endif + if (_memory_page_size < min_span_size) + _memory_page_size = min_span_size; + if (_memory_page_size > max_page_size) + _memory_page_size = max_page_size; + _memory_page_size_shift = 0; + size_t page_size_bit = _memory_page_size; + while (page_size_bit != 1) { + ++_memory_page_size_shift; + page_size_bit >>= 1; + } + _memory_page_size = ((size_t)1 << _memory_page_size_shift); + +#if RPMALLOC_CONFIGURABLE + if (!_memory_config.span_size) { + _memory_span_size = _memory_default_span_size; + _memory_span_size_shift = _memory_default_span_size_shift; + _memory_span_mask = _memory_default_span_mask; + } else { + size_t span_size = _memory_config.span_size; + if (span_size > (256 * 1024)) + span_size = (256 * 1024); + _memory_span_size = 4096; + _memory_span_size_shift = 12; + while (_memory_span_size < span_size) { + _memory_span_size <<= 1; + ++_memory_span_size_shift; + } + _memory_span_mask = ~(uintptr_t)(_memory_span_size - 1); + } +#endif + + _memory_span_map_count = ( _memory_config.span_map_count ? _memory_config.span_map_count : DEFAULT_SPAN_MAP_COUNT); + if ((_memory_span_size * _memory_span_map_count) < _memory_page_size) + _memory_span_map_count = (_memory_page_size / _memory_span_size); + if ((_memory_page_size >= _memory_span_size) && ((_memory_span_map_count * _memory_span_size) % _memory_page_size)) + _memory_span_map_count = (_memory_page_size / _memory_span_size); + _memory_heap_reserve_count = (_memory_span_map_count > DEFAULT_SPAN_MAP_COUNT) ? DEFAULT_SPAN_MAP_COUNT : _memory_span_map_count; + + _memory_config.page_size = _memory_page_size; + _memory_config.span_size = _memory_span_size; + _memory_config.span_map_count = _memory_span_map_count; + _memory_config.enable_huge_pages = _memory_huge_pages; + +#if ((defined(__APPLE__) || defined(__HAIKU__)) && ENABLE_PRELOAD) || defined(__TINYC__) + if (pthread_key_create(&_memory_thread_heap, _rpmalloc_heap_release_raw_fc)) + return -1; +#endif +#if defined(_WIN32) && (!defined(BUILD_DYNAMIC_LINK) || !BUILD_DYNAMIC_LINK) + fls_key = FlsAlloc(&_rpmalloc_thread_destructor); +#endif + + //Setup all small and medium size classes + size_t iclass = 0; + _memory_size_class[iclass].block_size = SMALL_GRANULARITY; + _rpmalloc_adjust_size_class(iclass); + for (iclass = 1; iclass < SMALL_CLASS_COUNT; ++iclass) { + size_t size = iclass * SMALL_GRANULARITY; + _memory_size_class[iclass].block_size = (uint32_t)size; + _rpmalloc_adjust_size_class(iclass); + } + //At least two blocks per span, then fall back to large allocations + _memory_medium_size_limit = (_memory_span_size - SPAN_HEADER_SIZE) >> 1; + if (_memory_medium_size_limit > MEDIUM_SIZE_LIMIT) + _memory_medium_size_limit = MEDIUM_SIZE_LIMIT; + for (iclass = 0; iclass < MEDIUM_CLASS_COUNT; ++iclass) { + size_t size = SMALL_SIZE_LIMIT + ((iclass + 1) * MEDIUM_GRANULARITY); + if (size > _memory_medium_size_limit) + break; + _memory_size_class[SMALL_CLASS_COUNT + iclass].block_size = (uint32_t)size; + _rpmalloc_adjust_size_class(SMALL_CLASS_COUNT + iclass); + } + + _memory_orphan_heaps = 0; +#if RPMALLOC_FIRST_CLASS_HEAPS + _memory_first_class_orphan_heaps = 0; +#endif +#if ENABLE_STATISTICS + atomic_store32(&_memory_active_heaps, 0); + atomic_store32(&_mapped_pages, 0); + _mapped_pages_peak = 0; + atomic_store32(&_master_spans, 0); + atomic_store32(&_mapped_total, 0); + atomic_store32(&_unmapped_total, 0); + atomic_store32(&_mapped_pages_os, 0); + atomic_store32(&_huge_pages_current, 0); + _huge_pages_peak = 0; +#endif + memset(_memory_heaps, 0, sizeof(_memory_heaps)); + atomic_store32_release(&_memory_global_lock, 0); + + rpmalloc_linker_reference(); + + //Initialize this thread + rpmalloc_thread_initialize(); + return 0; +} + +//! Finalize the allocator +void +rpmalloc_finalize(void) { + rpmalloc_thread_finalize(1); + //rpmalloc_dump_statistics(stdout); + + if (_memory_global_reserve) { + atomic_add32(&_memory_global_reserve_master->remaining_spans, -(int32_t)_memory_global_reserve_count); + _memory_global_reserve_master = 0; + _memory_global_reserve_count = 0; + _memory_global_reserve = 0; + } + atomic_store32_release(&_memory_global_lock, 0); + + //Free all thread caches and fully free spans + for (size_t list_idx = 0; list_idx < HEAP_ARRAY_SIZE; ++list_idx) { + heap_t* heap = _memory_heaps[list_idx]; + while (heap) { + heap_t* next_heap = heap->next_heap; + heap->finalize = 1; + _rpmalloc_heap_global_finalize(heap); + heap = next_heap; + } + } + +#if ENABLE_GLOBAL_CACHE + //Free global caches + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) + _rpmalloc_global_cache_finalize(&_memory_span_cache[iclass]); +#endif + +#if (defined(__APPLE__) || defined(__HAIKU__)) && ENABLE_PRELOAD + pthread_key_delete(_memory_thread_heap); +#endif +#if defined(_WIN32) && (!defined(BUILD_DYNAMIC_LINK) || !BUILD_DYNAMIC_LINK) + FlsFree(fls_key); + fls_key = 0; +#endif +#if ENABLE_STATISTICS + //If you hit these asserts you probably have memory leaks (perhaps global scope data doing dynamic allocations) or double frees in your code + rpmalloc_assert(atomic_load32(&_mapped_pages) == 0, "Memory leak detected"); + rpmalloc_assert(atomic_load32(&_mapped_pages_os) == 0, "Memory leak detected"); +#endif + + _rpmalloc_initialized = 0; +} + +//! Initialize thread, assign heap +extern inline void +rpmalloc_thread_initialize(void) { + if (!get_thread_heap_raw()) { + heap_t* heap = _rpmalloc_heap_allocate(0); + if (heap) { + _rpmalloc_stat_inc(&_memory_active_heaps); + set_thread_heap(heap); +#if defined(_WIN32) && (!defined(BUILD_DYNAMIC_LINK) || !BUILD_DYNAMIC_LINK) + FlsSetValue(fls_key, heap); +#endif + } + } +} + +//! Finalize thread, orphan heap +void +rpmalloc_thread_finalize(int release_caches) { + heap_t* heap = get_thread_heap_raw(); + if (heap) + _rpmalloc_heap_release_raw(heap, release_caches); + set_thread_heap(0); +#if defined(_WIN32) && (!defined(BUILD_DYNAMIC_LINK) || !BUILD_DYNAMIC_LINK) + FlsSetValue(fls_key, 0); +#endif +} + +int +rpmalloc_is_thread_initialized(void) { + return (get_thread_heap_raw() != 0) ? 1 : 0; +} + +const rpmalloc_config_t* +rpmalloc_config(void) { + return &_memory_config; +} + +// Extern interface + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc(size_t size) { +#if ENABLE_VALIDATE_ARGS + if (size >= MAX_ALLOC_SIZE) { + errno = EINVAL; + return 0; + } +#endif + heap_t* heap = get_thread_heap(); + return _rpmalloc_allocate(heap, size); +} + +extern inline void +rpfree(void* ptr) { + _rpmalloc_deallocate(ptr); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpcalloc(size_t num, size_t size) { + size_t total; +#if ENABLE_VALIDATE_ARGS +#if PLATFORM_WINDOWS + int err = SizeTMult(num, size, &total); + if ((err != S_OK) || (total >= MAX_ALLOC_SIZE)) { + errno = EINVAL; + return 0; + } +#else + int err = __builtin_umull_overflow(num, size, &total); + if (err || (total >= MAX_ALLOC_SIZE)) { + errno = EINVAL; + return 0; + } +#endif +#else + total = num * size; +#endif + heap_t* heap = get_thread_heap(); + void* block = _rpmalloc_allocate(heap, total); + if (block) + memset(block, 0, total); + return block; +} + +extern inline RPMALLOC_ALLOCATOR void* +rprealloc(void* ptr, size_t size) { +#if ENABLE_VALIDATE_ARGS + if (size >= MAX_ALLOC_SIZE) { + errno = EINVAL; + return ptr; + } +#endif + heap_t* heap = get_thread_heap(); + return _rpmalloc_reallocate(heap, ptr, size, 0, 0); +} + +extern RPMALLOC_ALLOCATOR void* +rpaligned_realloc(void* ptr, size_t alignment, size_t size, size_t oldsize, + unsigned int flags) { +#if ENABLE_VALIDATE_ARGS + if ((size + alignment < size) || (alignment > _memory_page_size)) { + errno = EINVAL; + return 0; + } +#endif + heap_t* heap = get_thread_heap(); + return _rpmalloc_aligned_reallocate(heap, ptr, alignment, size, oldsize, flags); +} + +extern RPMALLOC_ALLOCATOR void* +rpaligned_alloc(size_t alignment, size_t size) { + heap_t* heap = get_thread_heap(); + return _rpmalloc_aligned_allocate(heap, alignment, size); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpaligned_calloc(size_t alignment, size_t num, size_t size) { + size_t total; +#if ENABLE_VALIDATE_ARGS +#if PLATFORM_WINDOWS + int err = SizeTMult(num, size, &total); + if ((err != S_OK) || (total >= MAX_ALLOC_SIZE)) { + errno = EINVAL; + return 0; + } +#else + int err = __builtin_umull_overflow(num, size, &total); + if (err || (total >= MAX_ALLOC_SIZE)) { + errno = EINVAL; + return 0; + } +#endif +#else + total = num * size; +#endif + void* block = rpaligned_alloc(alignment, total); + if (block) + memset(block, 0, total); + return block; +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmemalign(size_t alignment, size_t size) { + return rpaligned_alloc(alignment, size); +} + +extern inline int +rpposix_memalign(void **memptr, size_t alignment, size_t size) { + if (memptr) + *memptr = rpaligned_alloc(alignment, size); + else + return EINVAL; + return *memptr ? 0 : ENOMEM; +} + +extern inline size_t +rpmalloc_usable_size(void* ptr) { + return (ptr ? _rpmalloc_usable_size(ptr) : 0); +} + +extern inline void +rpmalloc_thread_collect(void) { +} + +void +rpmalloc_thread_statistics(rpmalloc_thread_statistics_t* stats) { + memset(stats, 0, sizeof(rpmalloc_thread_statistics_t)); + heap_t* heap = get_thread_heap_raw(); + if (!heap) + return; + + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + size_class_t* size_class = _memory_size_class + iclass; + span_t* span = heap->size_class[iclass].partial_span; + while (span) { + size_t free_count = span->list_size; + size_t block_count = size_class->block_count; + if (span->free_list_limit < block_count) + block_count = span->free_list_limit; + free_count += (block_count - span->used_count); + stats->sizecache = free_count * size_class->block_size; + span = span->next; + } + } + +#if ENABLE_THREAD_CACHE + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + span_cache_t* span_cache; + if (!iclass) + span_cache = &heap->span_cache; + else + span_cache = (span_cache_t*)(heap->span_large_cache + (iclass - 1)); + stats->spancache = span_cache->count * (iclass + 1) * _memory_span_size; + } +#endif + + span_t* deferred = (span_t*)atomic_load_ptr(&heap->span_free_deferred); + while (deferred) { + if (deferred->size_class != SIZE_CLASS_HUGE) + stats->spancache = (size_t)deferred->span_count * _memory_span_size; + deferred = (span_t*)deferred->free_list; + } + +#if ENABLE_STATISTICS + stats->thread_to_global = (size_t)atomic_load64(&heap->thread_to_global); + stats->global_to_thread = (size_t)atomic_load64(&heap->global_to_thread); + + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + stats->span_use[iclass].current = (size_t)atomic_load32(&heap->span_use[iclass].current); + stats->span_use[iclass].peak = (size_t)atomic_load32(&heap->span_use[iclass].high); + stats->span_use[iclass].to_global = (size_t)atomic_load32(&heap->span_use[iclass].spans_to_global); + stats->span_use[iclass].from_global = (size_t)atomic_load32(&heap->span_use[iclass].spans_from_global); + stats->span_use[iclass].to_cache = (size_t)atomic_load32(&heap->span_use[iclass].spans_to_cache); + stats->span_use[iclass].from_cache = (size_t)atomic_load32(&heap->span_use[iclass].spans_from_cache); + stats->span_use[iclass].to_reserved = (size_t)atomic_load32(&heap->span_use[iclass].spans_to_reserved); + stats->span_use[iclass].from_reserved = (size_t)atomic_load32(&heap->span_use[iclass].spans_from_reserved); + stats->span_use[iclass].map_calls = (size_t)atomic_load32(&heap->span_use[iclass].spans_map_calls); + } + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + stats->size_use[iclass].alloc_current = (size_t)atomic_load32(&heap->size_class_use[iclass].alloc_current); + stats->size_use[iclass].alloc_peak = (size_t)heap->size_class_use[iclass].alloc_peak; + stats->size_use[iclass].alloc_total = (size_t)atomic_load32(&heap->size_class_use[iclass].alloc_total); + stats->size_use[iclass].free_total = (size_t)atomic_load32(&heap->size_class_use[iclass].free_total); + stats->size_use[iclass].spans_to_cache = (size_t)atomic_load32(&heap->size_class_use[iclass].spans_to_cache); + stats->size_use[iclass].spans_from_cache = (size_t)atomic_load32(&heap->size_class_use[iclass].spans_from_cache); + stats->size_use[iclass].spans_from_reserved = (size_t)atomic_load32(&heap->size_class_use[iclass].spans_from_reserved); + stats->size_use[iclass].map_calls = (size_t)atomic_load32(&heap->size_class_use[iclass].spans_map_calls); + } +#endif +} + +void +rpmalloc_global_statistics(rpmalloc_global_statistics_t* stats) { + memset(stats, 0, sizeof(rpmalloc_global_statistics_t)); +#if ENABLE_STATISTICS + stats->mapped = (size_t)atomic_load32(&_mapped_pages) * _memory_page_size; + stats->mapped_peak = (size_t)_mapped_pages_peak * _memory_page_size; + stats->mapped_total = (size_t)atomic_load32(&_mapped_total) * _memory_page_size; + stats->unmapped_total = (size_t)atomic_load32(&_unmapped_total) * _memory_page_size; + stats->huge_alloc = (size_t)atomic_load32(&_huge_pages_current) * _memory_page_size; + stats->huge_alloc_peak = (size_t)_huge_pages_peak * _memory_page_size; +#endif +#if ENABLE_GLOBAL_CACHE + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) + stats->cached += _memory_span_cache[iclass].count * (iclass + 1) * _memory_span_size; +#endif +} + +#if ENABLE_STATISTICS + +static void +_memory_heap_dump_statistics(heap_t* heap, void* file) { + fprintf(file, "Heap %d stats:\n", heap->id); + fprintf(file, "Class CurAlloc PeakAlloc TotAlloc TotFree BlkSize BlkCount SpansCur SpansPeak PeakAllocMiB ToCacheMiB FromCacheMiB FromReserveMiB MmapCalls\n"); + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + if (!atomic_load32(&heap->size_class_use[iclass].alloc_total)) + continue; + fprintf(file, "%3u: %10u %10u %10u %10u %8u %8u %8d %9d %13zu %11zu %12zu %14zu %9u\n", (uint32_t)iclass, + atomic_load32(&heap->size_class_use[iclass].alloc_current), + heap->size_class_use[iclass].alloc_peak, + atomic_load32(&heap->size_class_use[iclass].alloc_total), + atomic_load32(&heap->size_class_use[iclass].free_total), + _memory_size_class[iclass].block_size, + _memory_size_class[iclass].block_count, + atomic_load32(&heap->size_class_use[iclass].spans_current), + heap->size_class_use[iclass].spans_peak, + ((size_t)heap->size_class_use[iclass].alloc_peak * (size_t)_memory_size_class[iclass].block_size) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->size_class_use[iclass].spans_to_cache) * _memory_span_size) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->size_class_use[iclass].spans_from_cache) * _memory_span_size) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->size_class_use[iclass].spans_from_reserved) * _memory_span_size) / (size_t)(1024 * 1024), + atomic_load32(&heap->size_class_use[iclass].spans_map_calls)); + } + fprintf(file, "Spans Current Peak Deferred PeakMiB Cached ToCacheMiB FromCacheMiB ToReserveMiB FromReserveMiB ToGlobalMiB FromGlobalMiB MmapCalls\n"); + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + if (!atomic_load32(&heap->span_use[iclass].high) && !atomic_load32(&heap->span_use[iclass].spans_map_calls)) + continue; + fprintf(file, "%4u: %8d %8u %8u %8zu %7u %11zu %12zu %12zu %14zu %11zu %13zu %10u\n", (uint32_t)(iclass + 1), + atomic_load32(&heap->span_use[iclass].current), + atomic_load32(&heap->span_use[iclass].high), + atomic_load32(&heap->span_use[iclass].spans_deferred), + ((size_t)atomic_load32(&heap->span_use[iclass].high) * (size_t)_memory_span_size * (iclass + 1)) / (size_t)(1024 * 1024), +#if ENABLE_THREAD_CACHE + (unsigned int)(!iclass ? heap->span_cache.count : heap->span_large_cache[iclass - 1].count), + ((size_t)atomic_load32(&heap->span_use[iclass].spans_to_cache) * (iclass + 1) * _memory_span_size) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->span_use[iclass].spans_from_cache) * (iclass + 1) * _memory_span_size) / (size_t)(1024 * 1024), +#else + 0, (size_t)0, (size_t)0, +#endif + ((size_t)atomic_load32(&heap->span_use[iclass].spans_to_reserved) * (iclass + 1) * _memory_span_size) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->span_use[iclass].spans_from_reserved) * (iclass + 1) * _memory_span_size) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->span_use[iclass].spans_to_global) * (size_t)_memory_span_size * (iclass + 1)) / (size_t)(1024 * 1024), + ((size_t)atomic_load32(&heap->span_use[iclass].spans_from_global) * (size_t)_memory_span_size * (iclass + 1)) / (size_t)(1024 * 1024), + atomic_load32(&heap->span_use[iclass].spans_map_calls)); + } + fprintf(file, "Full spans: %zu\n", heap->full_span_count); + fprintf(file, "ThreadToGlobalMiB GlobalToThreadMiB\n"); + fprintf(file, "%17zu %17zu\n", (size_t)atomic_load64(&heap->thread_to_global) / (size_t)(1024 * 1024), (size_t)atomic_load64(&heap->global_to_thread) / (size_t)(1024 * 1024)); +} + +#endif + +void +rpmalloc_dump_statistics(void* file) { +#if ENABLE_STATISTICS + for (size_t list_idx = 0; list_idx < HEAP_ARRAY_SIZE; ++list_idx) { + heap_t* heap = _memory_heaps[list_idx]; + while (heap) { + int need_dump = 0; + for (size_t iclass = 0; !need_dump && (iclass < SIZE_CLASS_COUNT); ++iclass) { + if (!atomic_load32(&heap->size_class_use[iclass].alloc_total)) { + rpmalloc_assert(!atomic_load32(&heap->size_class_use[iclass].free_total), "Heap statistics counter mismatch"); + rpmalloc_assert(!atomic_load32(&heap->size_class_use[iclass].spans_map_calls), "Heap statistics counter mismatch"); + continue; + } + need_dump = 1; + } + for (size_t iclass = 0; !need_dump && (iclass < LARGE_CLASS_COUNT); ++iclass) { + if (!atomic_load32(&heap->span_use[iclass].high) && !atomic_load32(&heap->span_use[iclass].spans_map_calls)) + continue; + need_dump = 1; + } + if (need_dump) + _memory_heap_dump_statistics(heap, file); + heap = heap->next_heap; + } + } + fprintf(file, "Global stats:\n"); + size_t huge_current = (size_t)atomic_load32(&_huge_pages_current) * _memory_page_size; + size_t huge_peak = (size_t)_huge_pages_peak * _memory_page_size; + fprintf(file, "HugeCurrentMiB HugePeakMiB\n"); + fprintf(file, "%14zu %11zu\n", huge_current / (size_t)(1024 * 1024), huge_peak / (size_t)(1024 * 1024)); + + fprintf(file, "GlobalCacheMiB\n"); + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + global_cache_t* cache = _memory_span_cache + iclass; + size_t global_cache = (size_t)cache->count * iclass * _memory_span_size; + + size_t global_overflow_cache = 0; + span_t* span = cache->overflow; + while (span) { + global_overflow_cache += iclass * _memory_span_size; + span = span->next; + } + if (global_cache || global_overflow_cache || cache->insert_count || cache->extract_count) + fprintf(file, "%4zu: %8zuMiB (%8zuMiB overflow) %14zu insert %14zu extract\n", iclass + 1, global_cache / (size_t)(1024 * 1024), global_overflow_cache / (size_t)(1024 * 1024), cache->insert_count, cache->extract_count); + } + + size_t mapped = (size_t)atomic_load32(&_mapped_pages) * _memory_page_size; + size_t mapped_os = (size_t)atomic_load32(&_mapped_pages_os) * _memory_page_size; + size_t mapped_peak = (size_t)_mapped_pages_peak * _memory_page_size; + size_t mapped_total = (size_t)atomic_load32(&_mapped_total) * _memory_page_size; + size_t unmapped_total = (size_t)atomic_load32(&_unmapped_total) * _memory_page_size; + fprintf(file, "MappedMiB MappedOSMiB MappedPeakMiB MappedTotalMiB UnmappedTotalMiB\n"); + fprintf(file, "%9zu %11zu %13zu %14zu %16zu\n", + mapped / (size_t)(1024 * 1024), + mapped_os / (size_t)(1024 * 1024), + mapped_peak / (size_t)(1024 * 1024), + mapped_total / (size_t)(1024 * 1024), + unmapped_total / (size_t)(1024 * 1024)); + + fprintf(file, "\n"); +#if 0 + int64_t allocated = atomic_load64(&_allocation_counter); + int64_t deallocated = atomic_load64(&_deallocation_counter); + fprintf(file, "Allocation count: %lli\n", allocated); + fprintf(file, "Deallocation count: %lli\n", deallocated); + fprintf(file, "Current allocations: %lli\n", (allocated - deallocated)); + fprintf(file, "Master spans: %d\n", atomic_load32(&_master_spans)); + fprintf(file, "Dangling master spans: %d\n", atomic_load32(&_unmapped_master_spans)); +#endif +#endif + (void)sizeof(file); +} + +#if RPMALLOC_FIRST_CLASS_HEAPS + +extern inline rpmalloc_heap_t* +rpmalloc_heap_acquire(void) { + // Must be a pristine heap from newly mapped memory pages, or else memory blocks + // could already be allocated from the heap which would (wrongly) be released when + // heap is cleared with rpmalloc_heap_free_all(). Also heaps guaranteed to be + // pristine from the dedicated orphan list can be used. + heap_t* heap = _rpmalloc_heap_allocate(1); + heap->owner_thread = 0; + _rpmalloc_stat_inc(&_memory_active_heaps); + return heap; +} + +extern inline void +rpmalloc_heap_release(rpmalloc_heap_t* heap) { + if (heap) + _rpmalloc_heap_release(heap, 1, 1); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc_heap_alloc(rpmalloc_heap_t* heap, size_t size) { +#if ENABLE_VALIDATE_ARGS + if (size >= MAX_ALLOC_SIZE) { + errno = EINVAL; + return 0; + } +#endif + return _rpmalloc_allocate(heap, size); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc_heap_aligned_alloc(rpmalloc_heap_t* heap, size_t alignment, size_t size) { +#if ENABLE_VALIDATE_ARGS + if (size >= MAX_ALLOC_SIZE) { + errno = EINVAL; + return 0; + } +#endif + return _rpmalloc_aligned_allocate(heap, alignment, size); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc_heap_calloc(rpmalloc_heap_t* heap, size_t num, size_t size) { + return rpmalloc_heap_aligned_calloc(heap, 0, num, size); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc_heap_aligned_calloc(rpmalloc_heap_t* heap, size_t alignment, size_t num, size_t size) { + size_t total; +#if ENABLE_VALIDATE_ARGS +#if PLATFORM_WINDOWS + int err = SizeTMult(num, size, &total); + if ((err != S_OK) || (total >= MAX_ALLOC_SIZE)) { + errno = EINVAL; + return 0; + } +#else + int err = __builtin_umull_overflow(num, size, &total); + if (err || (total >= MAX_ALLOC_SIZE)) { + errno = EINVAL; + return 0; + } +#endif +#else + total = num * size; +#endif + void* block = _rpmalloc_aligned_allocate(heap, alignment, total); + if (block) + memset(block, 0, total); + return block; +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc_heap_realloc(rpmalloc_heap_t* heap, void* ptr, size_t size, unsigned int flags) { +#if ENABLE_VALIDATE_ARGS + if (size >= MAX_ALLOC_SIZE) { + errno = EINVAL; + return ptr; + } +#endif + return _rpmalloc_reallocate(heap, ptr, size, 0, flags); +} + +extern inline RPMALLOC_ALLOCATOR void* +rpmalloc_heap_aligned_realloc(rpmalloc_heap_t* heap, void* ptr, size_t alignment, size_t size, unsigned int flags) { +#if ENABLE_VALIDATE_ARGS + if ((size + alignment < size) || (alignment > _memory_page_size)) { + errno = EINVAL; + return 0; + } +#endif + return _rpmalloc_aligned_reallocate(heap, ptr, alignment, size, 0, flags); +} + +extern inline void +rpmalloc_heap_free(rpmalloc_heap_t* heap, void* ptr) { + (void)sizeof(heap); + _rpmalloc_deallocate(ptr); +} + +extern inline void +rpmalloc_heap_free_all(rpmalloc_heap_t* heap) { + span_t* span; + span_t* next_span; + + _rpmalloc_heap_cache_adopt_deferred(heap, 0); + + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + span = heap->size_class[iclass].partial_span; + while (span) { + next_span = span->next; + _rpmalloc_heap_cache_insert(heap, span); + span = next_span; + } + heap->size_class[iclass].partial_span = 0; + span = heap->full_span[iclass]; + while (span) { + next_span = span->next; + _rpmalloc_heap_cache_insert(heap, span); + span = next_span; + } + } + memset(heap->size_class, 0, sizeof(heap->size_class)); + memset(heap->full_span, 0, sizeof(heap->full_span)); + + span = heap->large_huge_span; + while (span) { + next_span = span->next; + if (UNEXPECTED(span->size_class == SIZE_CLASS_HUGE)) + _rpmalloc_deallocate_huge(span); + else + _rpmalloc_heap_cache_insert(heap, span); + span = next_span; + } + heap->large_huge_span = 0; + heap->full_span_count = 0; + +#if ENABLE_THREAD_CACHE + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + span_cache_t* span_cache; + if (!iclass) + span_cache = &heap->span_cache; + else + span_cache = (span_cache_t*)(heap->span_large_cache + (iclass - 1)); + if (!span_cache->count) + continue; +#if ENABLE_GLOBAL_CACHE + _rpmalloc_stat_add64(&heap->thread_to_global, span_cache->count * (iclass + 1) * _memory_span_size); + _rpmalloc_stat_add(&heap->span_use[iclass].spans_to_global, span_cache->count); + _rpmalloc_global_cache_insert_spans(span_cache->span, iclass + 1, span_cache->count); +#else + for (size_t ispan = 0; ispan < span_cache->count; ++ispan) + _rpmalloc_span_unmap(span_cache->span[ispan]); +#endif + span_cache->count = 0; + } +#endif + +#if ENABLE_STATISTICS + for (size_t iclass = 0; iclass < SIZE_CLASS_COUNT; ++iclass) { + atomic_store32(&heap->size_class_use[iclass].alloc_current, 0); + atomic_store32(&heap->size_class_use[iclass].spans_current, 0); + } + for (size_t iclass = 0; iclass < LARGE_CLASS_COUNT; ++iclass) { + atomic_store32(&heap->span_use[iclass].current, 0); + } +#endif +} + +extern inline void +rpmalloc_heap_thread_set_current(rpmalloc_heap_t* heap) { + heap_t* prev_heap = get_thread_heap_raw(); + if (prev_heap != heap) { + set_thread_heap(heap); + if (prev_heap) + rpmalloc_heap_release(prev_heap); + } +} + +#endif + +#if ENABLE_PRELOAD || ENABLE_OVERRIDE + +#include "malloc.c" + +#endif + +void +rpmalloc_linker_reference(void) { + (void)sizeof(_rpmalloc_initialized); +} diff --git a/WinRpMalloc/src/rpmalloc.h b/WinRpMalloc/src/rpmalloc.h new file mode 100644 index 0000000..3806653 --- /dev/null +++ b/WinRpMalloc/src/rpmalloc.h @@ -0,0 +1,393 @@ +/* +* Copyright (c) 2022 Vaughn Nugent +* +* Library: VNLib +* Package: WinRpMalloc +* File: rpmalloc.h +* +* rpmalloc.h is part of WinRpMalloc which is part of the larger +* VNLib collection of libraries and utilities. +* +* WinRpMalloc is free software: you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published +* by the Free Software Foundation, either version 2 of the License, +* or (at your option) any later version. +* +* WinRpMalloc is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with WinRpMalloc. If not, see http://www.gnu.org/licenses/. +*/ + +/* rpmalloc.h - Memory allocator - Public Domain - 2016 Mattias Jansson + * + * This library provides a cross-platform lock free thread caching malloc implementation in C11. + * The latest source code is always available at + * + * https://github.com/mjansson/rpmalloc + * + * This library is put in the public domain; you can redistribute it and/or modify it without any restrictions. + * + */ + +#pragma once + +#include <stddef.h> + +#ifdef __cplusplus +extern "C" { +#endif + +#if defined(__clang__) || defined(__GNUC__) +# define RPMALLOC_EXPORT __attribute__((visibility("default"))) +# define RPMALLOC_ALLOCATOR +# if (defined(__clang_major__) && (__clang_major__ < 4)) || (defined(__GNUC__) && defined(ENABLE_PRELOAD) && ENABLE_PRELOAD) +# define RPMALLOC_ATTRIB_MALLOC +# define RPMALLOC_ATTRIB_ALLOC_SIZE(size) +# define RPMALLOC_ATTRIB_ALLOC_SIZE2(count, size) +# else +# define RPMALLOC_ATTRIB_MALLOC __attribute__((__malloc__)) +# define RPMALLOC_ATTRIB_ALLOC_SIZE(size) __attribute__((alloc_size(size))) +# define RPMALLOC_ATTRIB_ALLOC_SIZE2(count, size) __attribute__((alloc_size(count, size))) +# endif +# define RPMALLOC_CDECL +#elif defined(_MSC_VER) +# define RPMALLOC_EXPORT __declspec(dllexport) +# define RPMALLOC_ALLOCATOR __declspec(allocator) __declspec(restrict) +# define RPMALLOC_ATTRIB_MALLOC +# define RPMALLOC_ATTRIB_ALLOC_SIZE(size) +# define RPMALLOC_ATTRIB_ALLOC_SIZE2(count,size) +# define RPMALLOC_CDECL __cdecl +#else +# define RPMALLOC_EXPORT +# define RPMALLOC_ALLOCATOR +# define RPMALLOC_ATTRIB_MALLOC +# define RPMALLOC_ATTRIB_ALLOC_SIZE(size) +# define RPMALLOC_ATTRIB_ALLOC_SIZE2(count,size) +# define RPMALLOC_CDECL +#endif + +//! Define RPMALLOC_CONFIGURABLE to enable configuring sizes. Will introduce +// a very small overhead due to some size calculations not being compile time constants +#ifndef RPMALLOC_CONFIGURABLE +#define RPMALLOC_CONFIGURABLE 0 +#endif + +//! Define RPMALLOC_FIRST_CLASS_HEAPS to enable heap based API (rpmalloc_heap_* functions). +// Will introduce a very small overhead to track fully allocated spans in heaps +#ifndef RPMALLOC_FIRST_CLASS_HEAPS +#define RPMALLOC_FIRST_CLASS_HEAPS 0 +#endif + +//! Flag to rpaligned_realloc to not preserve content in reallocation +#define RPMALLOC_NO_PRESERVE 1 +//! Flag to rpaligned_realloc to fail and return null pointer if grow cannot be done in-place, +// in which case the original pointer is still valid (just like a call to realloc which failes to allocate +// a new block). +#define RPMALLOC_GROW_OR_FAIL 2 + +typedef struct rpmalloc_global_statistics_t { + //! Current amount of virtual memory mapped, all of which might not have been committed (only if ENABLE_STATISTICS=1) + size_t mapped; + //! Peak amount of virtual memory mapped, all of which might not have been committed (only if ENABLE_STATISTICS=1) + size_t mapped_peak; + //! Current amount of memory in global caches for small and medium sizes (<32KiB) + size_t cached; + //! Current amount of memory allocated in huge allocations, i.e larger than LARGE_SIZE_LIMIT which is 2MiB by default (only if ENABLE_STATISTICS=1) + size_t huge_alloc; + //! Peak amount of memory allocated in huge allocations, i.e larger than LARGE_SIZE_LIMIT which is 2MiB by default (only if ENABLE_STATISTICS=1) + size_t huge_alloc_peak; + //! Total amount of memory mapped since initialization (only if ENABLE_STATISTICS=1) + size_t mapped_total; + //! Total amount of memory unmapped since initialization (only if ENABLE_STATISTICS=1) + size_t unmapped_total; +} rpmalloc_global_statistics_t; + +typedef struct rpmalloc_thread_statistics_t { + //! Current number of bytes available in thread size class caches for small and medium sizes (<32KiB) + size_t sizecache; + //! Current number of bytes available in thread span caches for small and medium sizes (<32KiB) + size_t spancache; + //! Total number of bytes transitioned from thread cache to global cache (only if ENABLE_STATISTICS=1) + size_t thread_to_global; + //! Total number of bytes transitioned from global cache to thread cache (only if ENABLE_STATISTICS=1) + size_t global_to_thread; + //! Per span count statistics (only if ENABLE_STATISTICS=1) + struct { + //! Currently used number of spans + size_t current; + //! High water mark of spans used + size_t peak; + //! Number of spans transitioned to global cache + size_t to_global; + //! Number of spans transitioned from global cache + size_t from_global; + //! Number of spans transitioned to thread cache + size_t to_cache; + //! Number of spans transitioned from thread cache + size_t from_cache; + //! Number of spans transitioned to reserved state + size_t to_reserved; + //! Number of spans transitioned from reserved state + size_t from_reserved; + //! Number of raw memory map calls (not hitting the reserve spans but resulting in actual OS mmap calls) + size_t map_calls; + } span_use[64]; + //! Per size class statistics (only if ENABLE_STATISTICS=1) + struct { + //! Current number of allocations + size_t alloc_current; + //! Peak number of allocations + size_t alloc_peak; + //! Total number of allocations + size_t alloc_total; + //! Total number of frees + size_t free_total; + //! Number of spans transitioned to cache + size_t spans_to_cache; + //! Number of spans transitioned from cache + size_t spans_from_cache; + //! Number of spans transitioned from reserved state + size_t spans_from_reserved; + //! Number of raw memory map calls (not hitting the reserve spans but resulting in actual OS mmap calls) + size_t map_calls; + } size_use[128]; +} rpmalloc_thread_statistics_t; + +typedef struct rpmalloc_config_t { + //! Map memory pages for the given number of bytes. The returned address MUST be + // aligned to the rpmalloc span size, which will always be a power of two. + // Optionally the function can store an alignment offset in the offset variable + // in case it performs alignment and the returned pointer is offset from the + // actual start of the memory region due to this alignment. The alignment offset + // will be passed to the memory unmap function. The alignment offset MUST NOT be + // larger than 65535 (storable in an uint16_t), if it is you must use natural + // alignment to shift it into 16 bits. If you set a memory_map function, you + // must also set a memory_unmap function or else the default implementation will + // be used for both. This function must be thread safe, it can be called by + // multiple threads simultaneously. + void* (*memory_map)(size_t size, size_t* offset); + //! Unmap the memory pages starting at address and spanning the given number of bytes. + // If release is set to non-zero, the unmap is for an entire span range as returned by + // a previous call to memory_map and that the entire range should be released. The + // release argument holds the size of the entire span range. If release is set to 0, + // the unmap is a partial decommit of a subset of the mapped memory range. + // If you set a memory_unmap function, you must also set a memory_map function or + // else the default implementation will be used for both. This function must be thread + // safe, it can be called by multiple threads simultaneously. + void (*memory_unmap)(void* address, size_t size, size_t offset, size_t release); + //! Called when an assert fails, if asserts are enabled. Will use the standard assert() + // if this is not set. + void (*error_callback)(const char* message); + //! Called when a call to map memory pages fails (out of memory). If this callback is + // not set or returns zero the library will return a null pointer in the allocation + // call. If this callback returns non-zero the map call will be retried. The argument + // passed is the number of bytes that was requested in the map call. Only used if + // the default system memory map function is used (memory_map callback is not set). + int (*map_fail_callback)(size_t size); + //! Size of memory pages. The page size MUST be a power of two. All memory mapping + // requests to memory_map will be made with size set to a multiple of the page size. + // Used if RPMALLOC_CONFIGURABLE is defined to 1, otherwise system page size is used. + size_t page_size; + //! Size of a span of memory blocks. MUST be a power of two, and in [4096,262144] + // range (unless 0 - set to 0 to use the default span size). Used if RPMALLOC_CONFIGURABLE + // is defined to 1. + size_t span_size; + //! Number of spans to map at each request to map new virtual memory blocks. This can + // be used to minimize the system call overhead at the cost of virtual memory address + // space. The extra mapped pages will not be written until actually used, so physical + // committed memory should not be affected in the default implementation. Will be + // aligned to a multiple of spans that match memory page size in case of huge pages. + size_t span_map_count; + //! Enable use of large/huge pages. If this flag is set to non-zero and page size is + // zero, the allocator will try to enable huge pages and auto detect the configuration. + // If this is set to non-zero and page_size is also non-zero, the allocator will + // assume huge pages have been configured and enabled prior to initializing the + // allocator. + // For Windows, see https://docs.microsoft.com/en-us/windows/desktop/memory/large-page-support + // For Linux, see https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt + int enable_huge_pages; + //! Respectively allocated pages and huge allocated pages names for systems + // supporting it to be able to distinguish among anonymous regions. + const char *page_name; + const char *huge_page_name; +} rpmalloc_config_t; + +//! Initialize allocator with default configuration +RPMALLOC_EXPORT int +rpmalloc_initialize(void); + +//! Initialize allocator with given configuration +RPMALLOC_EXPORT int +rpmalloc_initialize_config(const rpmalloc_config_t* config); + +//! Get allocator configuration +RPMALLOC_EXPORT const rpmalloc_config_t* +rpmalloc_config(void); + +//! Finalize allocator +RPMALLOC_EXPORT void +rpmalloc_finalize(void); + +//! Initialize allocator for calling thread +RPMALLOC_EXPORT void +rpmalloc_thread_initialize(void); + +//! Finalize allocator for calling thread +RPMALLOC_EXPORT void +rpmalloc_thread_finalize(int release_caches); + +//! Perform deferred deallocations pending for the calling thread heap +RPMALLOC_EXPORT void +rpmalloc_thread_collect(void); + +//! Query if allocator is initialized for calling thread +RPMALLOC_EXPORT int +rpmalloc_is_thread_initialized(void); + +//! Get per-thread statistics +RPMALLOC_EXPORT void +rpmalloc_thread_statistics(rpmalloc_thread_statistics_t* stats); + +//! Get global statistics +RPMALLOC_EXPORT void +rpmalloc_global_statistics(rpmalloc_global_statistics_t* stats); + +//! Dump all statistics in human readable format to file (should be a FILE*) +RPMALLOC_EXPORT void +rpmalloc_dump_statistics(void* file); + +//! Allocate a memory block of at least the given size +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc(size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(1); + +//! Free the given memory block +RPMALLOC_EXPORT void +rpfree(void* ptr); + +//! Allocate a memory block of at least the given size and zero initialize it +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpcalloc(size_t num, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE2(1, 2); + +//! Reallocate the given block to at least the given size +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rprealloc(void* ptr, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(2); + +//! Reallocate the given block to at least the given size and alignment, +// with optional control flags (see RPMALLOC_NO_PRESERVE). +// Alignment must be a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB) +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpaligned_realloc(void* ptr, size_t alignment, size_t size, size_t oldsize, unsigned int flags) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(3); + +//! Allocate a memory block of at least the given size and alignment. +// Alignment must be a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB) +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpaligned_alloc(size_t alignment, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(2); + +//! Allocate a memory block of at least the given size and alignment, and zero initialize it. +// Alignment must be a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB) +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpaligned_calloc(size_t alignment, size_t num, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE2(2, 3); + +//! Allocate a memory block of at least the given size and alignment. +// Alignment must be a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB) +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmemalign(size_t alignment, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(2); + +//! Allocate a memory block of at least the given size and alignment. +// Alignment must be a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB) +RPMALLOC_EXPORT int +rpposix_memalign(void** memptr, size_t alignment, size_t size); + +//! Query the usable size of the given memory block (from given pointer to the end of block) +RPMALLOC_EXPORT size_t +rpmalloc_usable_size(void* ptr); + +//! Dummy empty function for forcing linker symbol inclusion +RPMALLOC_EXPORT void +rpmalloc_linker_reference(void); + +#if RPMALLOC_FIRST_CLASS_HEAPS + +//! Heap type +typedef struct heap_t rpmalloc_heap_t; + +//! Acquire a new heap. Will reuse existing released heaps or allocate memory for a new heap +// if none available. Heap API is implemented with the strict assumption that only one single +// thread will call heap functions for a given heap at any given time, no functions are thread safe. +RPMALLOC_EXPORT rpmalloc_heap_t* +rpmalloc_heap_acquire(void); + +//! Release a heap (does NOT free the memory allocated by the heap, use rpmalloc_heap_free_all before destroying the heap). +// Releasing a heap will enable it to be reused by other threads. Safe to pass a null pointer. +RPMALLOC_EXPORT void +rpmalloc_heap_release(rpmalloc_heap_t* heap); + +//! Allocate a memory block of at least the given size using the given heap. +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc_heap_alloc(rpmalloc_heap_t* heap, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(2); + +//! Allocate a memory block of at least the given size using the given heap. The returned +// block will have the requested alignment. Alignment must be a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB). +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc_heap_aligned_alloc(rpmalloc_heap_t* heap, size_t alignment, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(3); + +//! Allocate a memory block of at least the given size using the given heap and zero initialize it. +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc_heap_calloc(rpmalloc_heap_t* heap, size_t num, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE2(2, 3); + +//! Allocate a memory block of at least the given size using the given heap and zero initialize it. The returned +// block will have the requested alignment. Alignment must either be zero, or a power of two and a multiple of sizeof(void*), +// and should ideally be less than memory page size. A caveat of rpmalloc +// internals is that this must also be strictly less than the span size (default 64KiB). +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc_heap_aligned_calloc(rpmalloc_heap_t* heap, size_t alignment, size_t num, size_t size) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE2(2, 3); + +//! Reallocate the given block to at least the given size. The memory block MUST be allocated +// by the same heap given to this function. +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc_heap_realloc(rpmalloc_heap_t* heap, void* ptr, size_t size, unsigned int flags) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(3); + +//! Reallocate the given block to at least the given size. The memory block MUST be allocated +// by the same heap given to this function. The returned block will have the requested alignment. +// Alignment must be either zero, or a power of two and a multiple of sizeof(void*), and should ideally be +// less than memory page size. A caveat of rpmalloc internals is that this must also be strictly less than +// the span size (default 64KiB). +RPMALLOC_EXPORT RPMALLOC_ALLOCATOR void* +rpmalloc_heap_aligned_realloc(rpmalloc_heap_t* heap, void* ptr, size_t alignment, size_t size, unsigned int flags) RPMALLOC_ATTRIB_MALLOC RPMALLOC_ATTRIB_ALLOC_SIZE(4); + +//! Free the given memory block from the given heap. The memory block MUST be allocated +// by the same heap given to this function. +RPMALLOC_EXPORT void +rpmalloc_heap_free(rpmalloc_heap_t* heap, void* ptr); + +//! Free all memory allocated by the heap +RPMALLOC_EXPORT void +rpmalloc_heap_free_all(rpmalloc_heap_t* heap); + +//! Set the given heap as the current heap for the calling thread. A heap MUST only be current heap +// for a single thread, a heap can never be shared between multiple threads. The previous +// current heap for the calling thread is released to be reused by other threads. +RPMALLOC_EXPORT void +rpmalloc_heap_thread_set_current(rpmalloc_heap_t* heap); + +#endif + +#ifdef __cplusplus +} +#endif |