text:
[SOLVED] How do I get a consistent byte representation of strings in C# without manually specifying an encoding?
2009-01-23 13:39:54 8 Comments
How do I convert a
string to a
byte[] in .NET (C#) without manually specifying a specific encoding?
I'm going to encrypt the string. I can encrypt it without converting, but I'd still like to know why encoding comes to play here.
Also, why should encoding even be taken into consideration? Can't I simply get what bytes the string has been stored in? Why is there a dependency on character encodings?
30 comments
@jpmc26 2019-09-11 04:21:53
Upon being asked what you intend to do with the bytes, you responded:
I'm going to encrypt it. I can encrypt it without converting but I'd still like to know why encoding comes to play here. Just give me the bytes is what I say.
Regardless of whether you intend to send this encrypted data over the network, load it back into memory later, or steam it to another process, you are clearly intending to decrypt it at some point. In that case, the answer is that you're defining a communication protocol. A communication protocol should not be defined in terms of implementation details of your programming language and its associated runtime. There are several reasons for this:
- The program may be re-implemented in a different language or runtime in the future.
- The .NET implementation might change the internal representation of strings. You may think this sounds farfetched, but this actually happened in Java 9 to reduce memory usage. There's no reason .NET couldn't follow suit. Skeet suggests that UTF-16 probably isn't optimal today give the rise of the emoji and other blocks of Unicode needing more than 2 bytes to represent as well, increasing the likelihood that the internal representation could change in the future.
For communicating (either with a completely disparate process or with the same program in the future), you need to define your protocol strictly to minimize the difficulty of working with it or accidentally creating bugs. Depending on .NET's internal representation is not a strict, clear, or even guaranteed to be consistent definition. A standard encoding is a strict definition that will not fail you in the future.
In other words, you can't satisfy your requirement for consistency without specifying an encoding.
You may certainly choose to use UTF-16 directly if you find that your process performs significantly better since .NET uses it internally or for any other reason, but you need to choose that encoding explicitly and perform those conversions explicitly in your code rather than depending on .NET's internal implementation.
So choose an encoding and use it:
using System.Text;
// ...
Encoding.Unicode.GetBytes("abc"); # UTF-16 little endian
As you can see, it's also actually less code to just use the built in encoding objects than to implement your own reader/writer methods.
@user541686 2012-04-30 07:44:22
Contrary to the answers here, you DON'T need to worry about encoding if the bytes don't need to be interpreted!
Like you mentioned, your goal is, simply, to "get what bytes the string has been stored in".
(And, of course, to be able to re-construct the string from the bytes.)
For those goals, I honestly do not understand why people keep telling you that you need the encodings. You certainly do NOT need to worry about encodings for this.
Just do this instead:
static byte[] GetBytes(string str)
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
// Do NOT use on arbitrary bytes; only use on GetBytes's output on the SAME system
static string GetString(byte[] bytes)
char[] chars = new char[bytes.Length / sizeof(char)];
System.Buffer.BlockCopy(bytes, 0, chars, 0, bytes.Length);
return new string(chars);
As long as your program (or other programs) don't try to interpret the bytes somehow, which you obviously didn't mention you intend to do, then there is nothing wrong with this approach! Worrying about encodings just makes your life more complicated for no real reason.
Additional benefit to this approach:
It doesn't matter if the string contains invalid characters, because you can still get the data and reconstruct the original string anyway!
It will be encoded and decoded just the same, because you are just looking at the bytes.
If you used a specific encoding, though, it would've given you trouble with encoding/decoding invalid characters.
@Michael Buen 2012-04-30 09:11:29
+1 Exactly my thoughts, I don't know the insistence of some peeps here about encoding. Just need to do a memory dump / serialization(the default serialization library from Microsoft has flaws though). I hope I know this BlockCopy API before :-)
@user541686 2012-04-30 09:20:49
@MichaelBuen: Yup. As long as your memory dumps/serializations do not try to interpret the data, it's all fine. The rule of thumb to remember is this: If your program (or a different program) needs to convert the output of
GetBytes back to the same string, it may only use
GetString to do this. As long as you don't violate that, you can ignore the concept of encodings entirely.
@Ian1971 2012-05-11 11:16:46
@Mehrdad I agree with your logic, but I was surprised when I tested it that the encoding method is slightly faster. I guess I was expecting your method to be faster (there isn't much in it though)
@user541686 2012-05-11 13:29:05
@Ian1971: Might it be because
ToCharArray() allocates a new array, which gets subsequently discarded?
@Michael Buen 2012-05-13 11:06:57
@Ian1971 Encoding methods has its pitfalls though, it can't preserve the image copy of the original string; in particular, high surrogate characters can't be preserve with encoding method. Check this test: stackoverflow.com/a/10384024
@CodesInChaos 2012-05-13 11:14:52
What's ugly about this one is, that
GetString and
GetBytes need to executed on a system with the same endianness to work. So you can't use this to get bytes you want to turn into a string elsewhere. So I have a hard time to come up with a situations where I'd want to use this.
@Michael Buen 2012-05-13 11:25:45
@CodeInChaos just prefix a BOM before those bytes to indicate it came from .NET world(i.e. UTF-16) then mindprod.com/jgloss/utf.html
@user541686 2012-05-13 18:00:27
@CodeInChaos: Like I said, the whole point of this is if you want to use it on the same kind of system, with the same set of functions. If not, then you shouldn't use it.
@artbristol 2012-06-15 11:07:20
-1 I guarantee that someone (who doesn't understand bytes vs characters) is going to want to convert their string into a byte array, they will google it and read this answer, and they will do the wrong thing, because in almost all cases, the encoding IS relevant.
@user541686 2012-06-15 14:04:28
@artbristol: If they can't be bothered to read the answer (or the other answers...), then I'm sorry, then there's no better way for me to communicate with them. I generally opt for answering the OP rather than trying to guess what others might do with my answer -- the OP has the right to know, and just because someone might abuse a knife doesn't mean we need to hide all knives in the world for ourselves. Though if you disagree that's fine too.
@artbristol 2012-06-15 14:25:09
The question was asked 3 years ago, and is totally ambiguous. You have no evidence of how OP was going to use the bytes. Other people will have the exact same question, but will be planning to use the bytes in a situation where encoding matters, and your answer will be dead wrong in that case.
@user541686 2012-06-15 14:32:44
Well, the way I think about it is: I'm not a judge. I don't ask for "evidence" from the OP to try to prove his case before I answer him (contrary to what others might try to do). He clearly said, "Can't I simply get what bytes the string has been stored in? Why this dependency on encoding?", to which my answer is 100% accurate, more than the others on this page IMO. And IMO he's understood the caveats by now. Also, the fact that the answer was from 3 years ago is irrelevant. But again, if you'd rather ask for "evidence" first, then that's your style, and feel free to keep the downvote..
@David 2012-07-11 12:36:17
This answer is wrong on so many levels but foremost because of it's decleration "you DON'T need to worry about encoding!". The 2 methods, GetBytes and GetString are superfluous in as much as they are merely re-implementations of what Encoding.Unicode.GetBytes() and Encoding.Unicode.GetString() already do. The statement "As long as your program (or other programs) don't try to interpret the bytes" is also fundamentally flawed as implicitly they mean the bytes should be interpreted as Unicode.
@user541686 2012-07-11 15:04:19
@David: "...as implicitly they mean the bytes should be interpreted" I have no idea how you read the answer, but it "implicitly" means that they could be any encoding whatsoever. Also, if you think the methods are "merely reimplementations" of
Encoding.Unicode just because they do the same thing, then it seems like you're not understanding the abstraction layers correctly.
@David 2012-07-11 15:42:56
@Mehrdad ...it "implicitly" means that they could be any encoding whatsoever" I don't understand this statement, what exactly do you mean by this? As far as I can see, your
GetBytes() method will return a Unicode encoded byte array of a string and your
GetString() method will (if you pass a Unicode encoded byte array representation of a string) return a readable string and in any other encoding return garbage. Worse than that though
GetString() will crash if you pass it a UTF-8 encoded byte array of a string that contains an odd number of characters.
@user541686 2012-07-11 16:08:28
@David: Yes, it crashes on UTF-8 data, because
GetBytes never happens to return UTF-8 data. It seems like the abstraction layer you're expecting is different from the one that's actually there. If you're not sure how to use it correctly then don't; the answer probably wasn't intended for your use case. However, I 100% stand by my answer that it is correct for the use it was intended, which I tried to make perfectly clear.
@David 2012-07-11 16:33:10
@Mehrdad: Then we've come full circle then.
GetBytes and
GetString are re-implementations of
Encoding.Unicode.GetBytes()\GetString(). You are reframing your argument to side step your initial assertion of "any encoding whatsoever". I'm not disputing the code you've provided the OP wont work (for unicode at least) I just dont think it furthers his understanding of Encoding which he IS using however you try and hide it.
@user541686 2012-07-11 16:36:27
@David: Sigh, yes, they happen to be reimplementations, but that is irrelevant at this abstraction level. If you're even caring about that fact then you're using it wrong. If you don't know what I mean then please don't use it, but it's 100% valid for the OP's use case/abstraction level.
@k.c. 2012-10-29 10:27:37
I just need bytes for my crypto to work, i think you answer still rocks!
@Concrete Gannet 2013-02-25 01:05:17
-1 for the answer. +1 for David's and artbristol's comments above. Of course there is an in-memory representation of strings in .NET. It happens to be little endian UTF-16. When you get the byte array, you are getting them in that encoding. If all you ever want to do is convert from the byte array back to a string, the answer will suffice. But the answer is limited and dangerous. For example, if the bytes are to be included in an HTTP request, you need to know the encoding for the overall request. If you are in the business of converting characters to bytes, you must understand encoding.
@chiccodoro 2013-07-17 07:54:27
-1 for the answer, +1 for David's, artbristol's and Concrete's comments... This answer does NOT mention in any way that it only works if you execute both methods on the same platform. Furthermore it adds no value. The answer's argument is to provide a simple answer to a simple question but the answer is way more complicated than simply using the
Encoding.Unicode. You don't need to worry about encoding either if you simply use those methods, but they are safe no matter what platform you run them on.
@user541686 2013-08-01 08:15:51
@ConcreteGannet: I'm glad we both agree that "If all you ever want to do is convert from the byte array back to a string, the answer will suffice." That was the entire point of my answer.
@user541686 2013-08-01 08:26:00
@chiccodoro: Safety isn't the only concern here. On your (hypothetical?) system where UTF-16 isn't the internal representation,
Encoding.Unicode would be slower, with no benefit for the use cases this was intended for (which the OP has understood). Furthermore safety is only a a problem if you don't know what you're doing. You don't see C programmers avoiding pointers, despite of how "dangerous" they are, do you? You also don't see construction workers avoiding electric saws and drills. Just because you think something is dangerous doesn't mean people don't have a right to know about it.
@Concrete Gannet 2013-08-04 08:30:48
@Mehrdad: After some questioning, the OP says they intend to encrypt the string. In all likelihood, the next step after converting to a byte array would be some form of output. Whether your answer is correct or not depends on what is reading those encrypted bytes. The OP did not say a .NET application will read the encrypted bytes. If anything else is to read it, the OP should ensure the encoding is as expected by the reader. If the string is large and contains only or mostly plain ASCII, UTF-8 would be more compact, quicker to encrypt and quicker to output.
@Travis Watson 2013-08-05 21:36:03
Asking for the bytes of a
string in .NET is akin to asking for the bytes of a
object. The purpose of the
string and
char types is that the implementation details are abstracted. By using this answer, you are haphazardly circumventing implementation details and will be left with a fragile solution similar to binary serialization. There is no reason to use this answer, since using encoding is more robust, more portable, more logical, and most importantly easier. Seriously, the Encoding answers are one-liners... why do something crazy like this?!
@user541686 2013-08-05 21:59:55
@Travis: Except it's not the same thing as asking for the bytes of an
object: .NET specifically prevents you from doing that, but doesn't prevent you from doing this. That by itself should be enough to tell you there's a difference.
@Travis Watson 2013-08-05 22:40:55
@Mehrdad, digressing, you need to realize that technically possible does not equal pragmatically relevant nor architecturally sound. Back on topic, whether or not you realize it, you're effectively performing
System.Text.Encoding.Unicode.GetBytes(str) because that's what .NET is doing to represent the
string in memory. People are saying you don't understand Encoding because they know you can't avoid it. The only thing you're doing is jumping through hoops to hide it! Do you honestly still think this is a good idea?
@Travis Watson 2013-08-05 22:59:19
@Mehrdad, on a second read I noticed you ignored the entirety of my comment. The one part you did respond to, you misinterpreted (akin != same). I'm really starting to question why you're vehemently promoting this obviously flawed answer.
@user541686 2013-08-06 00:29:29
@Travis: I read your entire comment, but the entire basis of it was wrong (your claim that it's akin to reading the bytes of an
object). There's nothing similar between the two. What I'm telling you is that this code is meant for a different abstraction level than you think. Saying "it's just like
Encoding.Unicode.GetBytes" is wrong since it breaks that abstraction barrier. I don't know what else to tell you. My answer already served its purpose, which was to directly answer the OP's question. If you don't like my answer then downvote it; that's what it's for!
@user541686 2013-08-06 00:41:45
@Travis: The last thing I will tell you (because I just noticed it right now) is to read this answer below. I had already mentioned this before, but since that answer actually demonstrates it, I'll say it again: my answer saves & restores the string perfectly; encoding-based methods fail to work on
char sequences that can't be represented correctly.
@Kai Hartmann 2013-08-09 22:31:11
Isn't it that, because .NET uses UTF-16 internally and 16 bit characters therefore, a string in this example is in fact encoded using UTF-16? If you use Encoding.Unicode.GetString(), which is UTF-16, on the byte-array created in this example, it produces the original string value.
@Thomas Eding 2013-09-27 18:16:03
Yes, this answer works for niche use cases. But other answers work for all use cases. Why not use the superior (and just as easy to use... and less error prone) techniques that require you to type in an encoding? Giving this a big fat -1 because of that.
@user541686 2013-09-27 22:44:11
@Thomas: No, the other answers don't work for all use cases. Did you read Michael Buen's answer? His answer tells you why mine can handle cases that none of the other answers can. None of the answers here handles all cases, but mine handles the relevant cases to the OP.
@Thomas Eding 2013-09-27 22:56:54
@Mehrdad: Fair enough. But I still don't like this solution. Not sure completely about his (I don't feel like learning about unpaired surrogates at the moment, but it seems like it is something along the lines of a trap representation).
@user541686 2013-09-27 23:01:27
@Thomas: I don't really care if you "like" the solution (heck, I don't particularly either), but you can't deny that it's the only correct answer given here for the OP's use case (conversion between strings and byte arrays). The other answers destroy some
char sequences in the process, mine doesn't. Keep your downvote, but please think twice before hopping on the bandwagon and spreading misinformation.
@Thomas Eding 2013-09-27 23:03:49
How do I convert a string to a byte array in .NET (C#)? is the OP's described use case. Literally any answer that returns a
byte[] would be technically correct. But I'm done with this extended chat.
@BlueRaja - Danny Pflughoeft 2013-10-01 19:16:58
"Worrying about encodings just makes your life more complicated for no real reason." - Er, except that the answers that do worry about encoding are much simpler than this one. And of course, this answer still does rely on a particular encoding -
str.ToCharArray() must rely on an encoding, even if that encoding is not explicitly mentioned in the code (which can only be considered bad). I respect you a lot, Mehrdad, but this is a terrible answer.
@user541686 2013-10-01 19:47:01
@BlueRaja-DannyPflughoeft: Read my comments above. The abstraction layer we concern ourselves with here (i.e. the need for perfect 1:1 reconstruction on a given system) isn't the same as when you worry about encodings (i.e. interoperability with another system). They're two completely unrelated concerns and the former has nothing to do with encoding (and in fact cannot be done with any encoding scheme here).
@user1151923 2013-10-09 12:26:19
This does not keep the encoding intact. Too bad this is the accepted answer with the highest votes because I just wasted 2 hours trying to find out why my strings get garbled. Chased it down to a method that used this answer to convert string -> byte[].
@user541686 2013-10-09 19:29:30
@user1151923: Can you show me an example of an actual string that gets garbled? I can't fix the answer if you don't tell me how to reproduce the problem...
@user1151923 2013-10-10 13:43:47
var input = "тхис ис а тест"; var ms = new MemoryStream(GetBytes(input)); var sr = new StreamReader(ms); var output = sr.ReadToEnd(); output is B5AB
@user1151923 2013-10-10 13:49:20
I would add that I don't think "For those goals" is a justification to an answer that [sometimes] messes up encoding. What people are gonna see when they open this question is the question (.NET String to byte Array C#) and a highly rated answer claiming you don't need to worry about encoding in bold text (which by the way is missing the "for those goals" part). There are answers below that are shorter or just as long and that keep the encoding intact regardless of where and how you use the code.
@user541686 2013-10-10 18:11:44
@user1151923: Dude, the problem is with your code, not my answer! You're using
GetBytes to convert a string to bytes, but you're not using
GetString to go the reverse direction! These are supposed to be used in pairs; you can't just do whatever you feel like and expect it to work. If you don't use encodings one way you also have to ignore them in the reverse direction, but you ignored the fact that
StreamReader is encoding-based! Read my comment earlier: stackoverflow.com/questions/472906/…
@user541686 2013-10-10 18:26:12
@user1151923: And before you blame me for not warning you, realize that what has happened in your code is exactly equivalent to using
new StreamReader(stream).ReadToEnd() to go in one direction, but using
Encoding.UTF8 to go in the other direction. It's wrong because the writer was careless, and it has nothing to do with the answer that might have told you to use
UTF8. If the fact that
StreamReader uses UTF-16 by default is confusing, don't blame it on my answer; it's not my fault it was designed that way.
@Leon Lucardie 2013-10-23 10:48:49
@Mehrad Just because your answer is technically correct in this case doesn't make it a good answer for the reasons stated by many before me. It's like recommending the
goto statement when better alternatives are available because "well it works in this case doesn't it?". This site is meant for answers that will function properly for as many usecases as possible within the scope of the question. You announcing "YOU DONT NEED ENCODING" in a big size at the top of your answer while leaving the main caveat as a little side note at the bottom could lead to problems.
@user541686 2013-10-23 10:51:44
@LeonLucardie: The other alternatives aren't "better"; in fact, they're worse because they break on strings that can't be encoded correctly (such as those that contain unpaired surrogates). I've mentioned this a million times now, but apparently it's very convenient for people to ignore this fact...
@Chris Cirefice 2013-11-04 04:37:22
@Mehrdad Even in a perfect world where people would act professional, they won't take the time to do a little research. I'm almost positive that all pros and cons of this solution have been addressed in the comments here, as well as in the other answers. If there are still those who won't realize this fact and feel that continuing to argue (even 1.25 years later) over points that have already been addressed then it's not worth your time nor anyone else's to argue further. There are answers here that apply to both 'need-encoding' and 'don't-need-encoding' use cases; it's as simple as that.
@Jodrell 2014-04-07 08:43:55
+1, but, wouldn't
str.SelectMany(BitConvertor.GetBytes).ToArray(); suffice. (yes, I suspect
BlockCopy is faster.)
@user541686 2014-04-07 09:48:49
@Jodrell: You just answered yourself. And plus, it requires .NET 3.5 which should not be necessary.
@Steffen Winkler 2014-05-30 13:24:35
this is one of the worst pieces of code I've seen. And I saw people using DataTables in .NET 4! Neither the questioneer nor the person that posted this answer seem to understand what encoding actually means. Of course you are using encoding with this answer...but you don't know which encoding! Even if you are converting stuff on the same machine, who tells you that the user won't change his encoding, rendering the bytes unreadable?!
@Chris 2014-06-11 09:15:06
@SteffenWinkler: Yes, the answer does use an encoding but the point is that it doesn't care what. The reason is that it is guaranteed to be using the same encoding both ways. I'm not sure how you think that a user can change the encoding because this is the encoding that .NET uses to store strings. I don't believe a user could change it. If the runtime was changed then you'd be restarting the program so again both methods would be using the same encoding still.
@B. Clay Shannon 2014-06-11 18:38:25
This won't compile for me; the first line in the GetBytes() method fails with, "C:\Project\sscs\Handheld\Releases\6-4-0\HHS\PrintUtils.cs(752): sizeof can only be used in an unsafe context (consider using System.Runtime.InteropServices.Marshal.SizeOf)"
@user541686 2014-06-11 18:39:29
@B.ClayShannon: Are you on an old version of .NET? Just replace
sizeof(char) with
@B. Clay Shannon 2014-06-11 18:50:47
@Mehrdad: Yes, older than Rip Van Winkle. Just to use it, I have to use XP Mode and VS 2003; and yep, that allowed it to compile.
@Şafak Gür 2014-07-23 07:44:03
-1 It's scary how this is the accepted and highest voted answer. Yes, it may be useful to get the bytes of a string in the way it's stored on the memory. Yes, it may not matter the fact that it fails if
GetString and
GetBytes are called on machines with different endianness. But saying "you DON'T need to worry about encoding!" is so terrifyingly evil, since you encourage people to ignore The Absolute Minimum Every Software Developer Absolutely, Positively Must Know. @artbristol is right: The encoding IS relevant.
@user541686 2014-07-23 07:50:07
@ŞafakGür: Yes I do -- I do encourage people to ignore things that are irrelevant to the problem. What's really "evil" is teaching people to worry about the wrong thing. I believe the encoding is irrelevant to the question because the encoding is on an entirely different abstraction level. You obviously don't think so, so keep your downvote, and thanks for sharing your thoughts.
@Şafak Gür 2014-07-23 08:10:06
Don't get me wrong, simplicity is good. But the OP asked a very general question. Is he going to convert to and from the string on the same architecture? Is he going to write the bytes to a file and expect it to be viewed using a specific text editor? He didn't state any of these. So anyone who come to this question may read "You certainly do NOT need to worry about encodings" and think that encoding is not relevant nor needed, in any case. So if you said "Use this if you'll decode the bytes on the same machine and you don't need a specific encoding", this could be a great answer.
@user541686 2014-07-23 08:23:24
@ŞafakGür: You have to understand, the approach you want me to put into my answer is outright wrong because it's not bijective between the strings and byte arrays -- e.g., it doesn't preserve unpaired surrogates. I've said this a million times now. If it at least worked correctly, I would have considered it. But it doesn't -- it breaks on any string that doesn't happen to fit a Unicode encoding. That's why I insist so much on avoiding encodings altogether: they're not only unnecessary, they're blatantly wrong and don't work on arbitrary strings.
@user541686 2014-07-23 09:54:15
@artbristol: Well that's a new one. In all of your comments so far you never even once placed my understanding of the problem under question, and here you are now accusing me of being a brick wall who doesn't understand what String means in C#. For the record, I'm neither a brick wall nor a C programmer, which would have been fairly obvious if you looked at my badges in C vs. C# before pretending you knew me so well.
@tandztc 2014-09-02 09:28:13
this solution is not correct at all. The fatel error is that when bytes.Length is odd, the Length of chars is not enough to copy to, which rasie an ArgumentException says "Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection". We'd better use @bmotmans answer.
@user541686 2014-09-03 00:37:16
@tandztc: No, I think you're the one who's not using it correctly. How do you get an odd
bytes.Length in the first place? If you followed the answer correctly (which implies you're using
GetBytes to get the
bytes) then that event is impossible. If you got the byte array some other way then you have to convert it back to a string the same way instead of using this answer.
@tandztc 2014-09-03 04:37:10
@Mehrdad: Oh, sorry. I recognised that these two method must used in pairs. I misunderstood the usage because I'm searching for a solution of just convert byte array to string, so I leave a comment because the GetString method is not capable of handling all byte arrays. Sorry for bothering you again -:)
@g.pickardou 2014-10-10 14:11:55
-1 for the misleading declaration "you DON'T need to worry about encoding". This completely disregards the fact that algorithms mainly convert a string to a byte buffer because of some stream operation expects it. And when this serialization occurs, the encoding does matter either we serialize to a file or to a wire. Industry is throwing 1000s of workhours yearly because of encoding mismatch issues, the last thing we need is evangalizing "we do not need to worry about encoding..."
@supercat 2014-11-12 22:26:47
@Mehrdad: Would there be any objection to starting with an even-length
Byte[] and converting that to a string which can later be converted back to
Byte[]? I would think byte conversion would allow various "linear" operations (e.g. concatenating two strings produced by the conversion would be equivalent to converting the concatenation of the two arrays), while most other encodings would not. The only disadvantage I see with "straight" conversion is that the lexiographic ordering of the
String object would differ from that of the
byte[] [fixing that would require byte-swapping pairs].
@user541686 2014-11-12 22:48:27
@supercat: If you can guarantee it's even-length then no, but otherwise you'd lose the length information.
@supercat 2014-11-12 22:54:52
@Mehrdad: Perhaps it would be good to make clear that your method is appropriate for the predictable serialization of
String instances which might hold arbitrary binary data, as opposed to those which are known to hold valid UTF-16 strings. It's really too bad MS didn't include any other "immutable blob" types, since
String gets used oftentimes when some other standard blob type would probably be more appropriate if any existed.
@supercat 2014-11-13 18:00:06
@Mehrdad: Also, do you know of any nice way to convert a byte array to a string, with the bytes paired MSB-first, and preferably interpreting an odd-length array as though it was zero-padded? Using
String.CompareOrdinal on strings produced by converting the
KeyData from a
SortKey in such fashion will be faster than
SortKey.Compare, but producing such strings is a little slow.
@Erik A. Brandstadmoen 2014-11-26 11:40:01
I think you make assumptions on how strings are stored in the CLR. How do you know it is actually represented by a contiguous sequence of bytes? It might be represented as a linked list, or something else. Don't make assumptions. It will bite you in the umptions.
@user541686 2014-11-26 11:53:11
@ErikA.Brandstadmoen: Two things: (1) If it was anything other than a contiguous sequence of bytes then you couldn't obtain a pointer to the data in constant time via
fixed (char* p = str) { ... } (2) The reality is that this fact is actually 100% irrelevant because
ToCharArray always returns a char array regardless of the underlying data format, which is all we need and care about.
@Erik A. Brandstadmoen 2014-11-27 21:26:08
Of course, you are right, @Mehrdad. I read your answer too quickly. I thought you were charpointer-ing yourself into the string itself, which would of course just work if it is indeed represented by a contiguous byte array in memory. But, if you call
ToCharArray, the implementation of string storage is irrelevant, of course (except for efficiency...).
@F.Buster 2014-12-17 23:40:00
What makes this answer so horrible is the presumption that the OP just wants to "get the bytes" for some ephemeral operation, and then follows up with comments harping on the fact that using an encoding will destroy the invalid string by removing the unpaired surrogates. This begs the question, why is the data represented or stored as a string in the first place? A string is designed to represent text, not some broken or illegal sequence of characters. (continued ...)
@F.Buster 2014-12-17 23:40:48
Of course this roundabout pair of methods is technically correct because it satisfies some imaginary specifications for the OP's overwhelmingly under-specified use-case, but there are certainly more correct solutions for what the OP is actually trying to accomplish. Since we may never know what that may be, this answer is not only incorrect, but actively harmful as both an answer to this question and also in general.
@user541686 2014-12-18 00:25:22
@F.Buster: A string is designed to represent text, not some broken or illegal sequence of characters.... you're jumping to conclusions. Just because the string might not be valid UTF-16 doesn't mean it's "broken" or "not text". It just means you can't assume the encoding is UTF-16, so the answer needs to be independent of whatever encoding the string may happen to be using. And it is. If you don't like the question then I'm sorry, but this is the correct answer to the question.
@Greg D 2014-12-18 01:47:18
@Mehrdad: so the answer needs to be independent of whatever encoding the string may happen to be using <= This is conflating representation with abstraction. A string, as just a string, is already independent of whatever encoding the implementation is using under the covers. The very act of any transcription of the string "Hello world" to some byte sequence is utilizing an encoding by definition. The only thing accomplished by plugging one's ears, shouting "LA LA LA!", and reinterpreting a block of memory as bytes is hiding the encoding that happened to be used.
@Aardvark 2015-01-26 17:06:36
I edited this answer by removing the repetitive defensiveness over it's correctness. I also move the technical explanations of why this is correct together at the start of the answer. I also changed the emphasis around a bit. I think this goes a long way to solving the flamewar over the answer.
@user541686 2015-01-26 18:21:28
Aardvark, your edit wasn't bad, but I didn't really see the point of it (and I noticed a little bit of a grammar/capitalization typo), so I rolled it back... I do think the original was good enough, and it's how I wanted to phrase things, and I'd rather it not be edited. I think the discussion did have a benefit and should stay, because (1) it helped the readers realize that this answer can controversial in a shared codebase, and (2) it allowed me to emphasize why I think the answer is the correct approach. Anyway, the discussion has rather ended already, so don't worry about it.
@kayleeFrye_onDeck 2015-03-16 01:22:27
This is precisely what I was looking for. I needed something that could send and receive events for observer patterns for a tech demo that just uses a simple Console App, and the event messages were being sent and received as byte arrays, so i figured one of the better ways to show the functionality was just to make the message a regular old string. This wouldn't be too helpful for most stuff, but it was exactly what I needed! Thanks a ton :)
@Karl Stephen 2015-03-23 08:50:14
BEST ANSWER for my specific issue, Thank you ! Used this to track conversion glitches between encodings for diagnostic purposes, on the same machine, the same application, without network connexions. Just because most of us are afraid one will use this to serialize datas and use them across platforms/databases is NOT a valid reason to set this answer on fire. Used this specifically to avoid disastrous encoding results. That's why I like SO so much : you can get here answers for very specific and unusual tasks. For beginners about safe string-bytes conversion, go re-read MSDN.
@chris 2015-04-10 17:41:58
I use this solution for converting password strings to
byte[] before salting and hashing them. In this use case, I absolutely do not care about encoding at all. I don't even need to convert the resulting hash back to a string - for password validation I just directly compare the resulting hash
byte[]s. Very elegant and low-overhead solution for this particular use case. The flame war here is a fun read, though.
@Josep 2015-04-15 13:54:13
I can see this code crashing in such a simple case:
sizeof(char) == 2
byteArray.Length == 9 Then,
(byteArray.Length / sizeof(char)) == 4, The call to
BlockCopy throws an Exception because you are going out of bounds. I would rather use up a bit more space and go for the easy solution of using Base64 Encoding from the
System.Convert class.
@chris 2015-04-15 18:51:21
@Josep How on earth would the length of the
byte[] ever be an odd number when the
GetString method is used the way it is intended here? Also keep in mind that this is just example code. In my not so uncommon use case that I described in my comment before (password hashing), converting back from
byte[] to
string isn't even necessary.
@user541686 2015-04-15 18:54:31
@Josep: I'm glad your code crashes, because it's trying to tell you that you're using it wrong. Instead of trying to get around it, realize that this answer was only meant to solve a particular problem, which is different from yours, and hence you shouldn't be using it.
@Gerard ONeill 2015-08-18 16:24:57
This answer is wrong for when strings are not stored as UTF-16 or any fixed length encoding. Which means that Encoding does matter, even if it doesn't show up in the code. Because for UTF-8, you will introduce empty 'bytes'. This also assumes that a string's storage and GetBytes will return the same encoding -- If not, then you aren't returning the "String's bytes". Fortunately the OP just wants bytes, which this answer provides.
@Vincent 2015-08-27 06:21:56
Just use Encoding.Unicode.GetBytes(). The function posted in this answer is 2x slower than Unicode.GetBytes(). Tested in Release & x64 environment.
@Powerlord 2015-09-19 21:49:23
If you don't know why encoding is important, you'd better hope you never have to deal with IBM's EBCDIC, whose characters don't match up to standard ASCII.
@John Leidegren 2015-10-25 10:05:05
About that endianness remark. The only platform that is running .NET and is not little-endian is the Xbox360 and the XNA track (which was the main method of getting your .NET software on the Xbox360) has been discontinued. There are some variants of mono that do run on big-endian platforms but this is the exception rather than the rule.
@camerondm9 2015-11-04 04:16:05
@JohnLeidegren Not true! Microsoft is porting the .Net Framework to Linux, and Linux runs on some big-endian architectures. See here for an example.
@John Leidegren 2015-11-04 08:20:13
@camerondm9 I'm not disputing the fact that these platforms exist but you have to consider that the CoreCLR does not JIT anything but X64 assembler (which is little-endian). To my knowledge Microsoft is currently not in the process of adding support for any other architecture, certainly not IBM PowerPC simply because there is no market for it. I'm not saying it can't happen I'm saying it's not happening any time soon. Disregarding everything I've said so far you still have to ask your self whether it is likely that you code will be running on a big-endian architecture in the near future?
@camerondm9 2015-11-05 03:13:58
@JohnLeidegren Microsoft has a JIT engine for ARM, and ARM is bi-endian (implementation defined). It may not be highly likely, but if your code may run on mobile devices (or it's a library), you never know...
@user541686 2015-12-01 20:23:56
@NumLock: Self-documentation.
sizeof(char) isn't 1, it's 2. This is C#, not C.
@Matthew Mark Miller 2016-01-26 18:48:16
The reason this answer is wrong is that it is IMPOSSIBLE to map a sequence of glyphs to a sequence of bytes without encoding. It is, however, also true that this example works without direct use of Encoding objects. That's because it is secretly asserting the canonical encoding scheme for Strings -- Unicode 16 I beleive -- is correct for all decoding implementations. This is true for .NET but it wouldn't be for other languages or runtimes. It's important that users KNOW what they're doing here is exporting an (already encoded) internal representation, and not really avoiding encoding.
@AMissico 2016-02-10 21:50:20
>>don't try to interpret the bytes somehow<< Just viewing the bytes is a form of interpretation
@AnorZaken 2016-04-10 03:27:02
This code will do what was intended, but beyond that the theoretical arguments are mostly garbage, albeit in such a way that it wont matter (be observed) in practice. Don't forget that language and compiler are also abstractions (which seldom make hard guarantees about physical memory). The statement that the char-array is the internal representation is reaching, as is citing pointer-code as proof. A string can be observed as a char-array, and manipulating char-pointers can be observed as you say, but could trivially be implemented as syntactic sugar for another physical representation.
@Firo 2016-05-09 09:57:46
@chris "low overhead" this is more code and slower than Encoding.Unicode.GetString/Bytes; "... for password validation I just directly compare the resulting hash" this will fail if you compile this code for PC and Xbox360 to use the same password validation, since the hash will be different for the same passwort
@yoel halb 2016-05-26 16:36:26
@BlueRaja-DannyPflughoeft ToCharArray() does not rely on encoding, it is in the .Net source just a copy of the internal representation of the internal bytes of the string, thus getting the char array using ToCharArray() has the same effect as fixing a pointer to the private member m_firstChar of string
@BlueRaja - Danny Pflughoeft 2016-05-26 16:44:22
@yoelhalb: You cannot convert a string to a byte array without relying on a particular encoding, literally be definition. In this case, you're using the encoding used by "the internal representation [..] of the string".
@Sören Kuklau 2016-06-08 17:01:17
@yoelhalb: yes it does. Of course it does. Not only because of what Danny said, but also because the API doc specifically says: "Copies the characters in this instance to a Unicode character array." The internal representation happens to be Unicode (UTF-16), but that's an irrelevant implementation detail.
@user541686 2016-11-08 11:48:24
@BlueRaja-DannyPflughoeft: It just hit me that you (and a lot of others here) are having a grammar issue. Notice I wrote "you don't need to worry about encoding" and yoel said it "does not rely on encoding". There was no article preceding "encoding"! yoel did NOT say it doesn't rely on "an encoding". We only said you do not have to worry about encoding anything to extract the bytes. You seem to think we're claiming the string somehow doesn't already possess an encoding, which is obviously nuts, and not what we're saying. We're just saying encoding (as a verb) shouldn't occur here.
@BlueRaja - Danny Pflughoeft 2016-11-08 16:28:26
No, there is no confusion. Your answer correctly states that you don't need to worry about encoding if you don't plan to interpret the string, but there are exactly 0 cases where this could possibly be useful. Even your own suggestion ("reconstruct the string") relies on the internal encoding of the string not changing. Meanwhile beginners see this answer and falsely believe they don't have to worry about what encodings are. This answer is worse than wrong, because it's technically correct but extremely detrimental.
@user541686 2016-11-08 19:02:20
@BlueRaja-DannyPflughoeft: "here are exactly 0 cases where this could possibly be useful." I already explained that this works even when the string is not valid UTF-16, so it's useful to people in that case. If you don't personally find it useful you don't have to use it.
@BlueMonkMN 2017-01-17 20:27:55
@Mehrdad I've been waffling back and forth on the argument and I can see good points on both sides. But in the end, I wonder why would you ever need to convert a string with invalid characters, assuming that's the only reason you would use this solution. Shouldn't we desperately try to avoid converting bytes to a string that technically isn't valid? Shouldn't data like that be forced to remain as a byte array to avoid giving anyone the impression that there's valid character data there?
@user541686 2017-01-17 21:10:54
@BlueMonkMN: I think your mistake is that this is not a method for converting bytes to strings and back to bytes. It is a method for converting strings to bytes and back to strings. There is a very crucial difference here. If you're asking why the user even has a string with invalid characters, or why
string even allows that, then that's an entirely different question, and not something I can or will attempt to answer here. I'm just trying to provide an answer that doesn't depend on the string's encoding (if any).
@BlueMonkMN 2017-01-17 21:18:48
@Mehrdad That's my point: I don't know how you could end up with a string containing invalid characters without converting it from an array of bytes. Every other "proper" means of generating strings that I can think of would not allow this because it would go through an encoding or be generated by a process that cannot return an invalid character. So my expectation is that one can always assume that .NET strings contain only valid characters, unless they use code like that which you have provided.
@user541686 2017-01-17 21:26:52
@BlueMonkMN: "That's my point: I don't know how you could end up with a string containing invalid characters without converting it from an array of bytes." ...well here's one:
"\uD800" + "\uDC00" both of these strings are invalid but their concatenation is valid. Maybe you want to convert each one to bytes, transmit them, and convert them back and then concatenate. Maybe they were generated by similarly splitting up a valid string. There's a million ways you could end up with invalid strings...
@Kris Vandermotten 2017-04-28 13:57:42
OP doesn't state why he wants to "simply get the bytes", but my guess is that he assumes that
System.Text.Encoding.Unicode.GetBytes(); is doing some kind of expensive conversion that he wants to avoid. Unfortunately, what you propose here is less efficient due to the double copy. Also, Endianness is important. OP wants to encrypt the string. It's likely that he doesn't do that to keep the encrypted stryng in memory. It will be written to disk or transferred across the network. What if it is to be decrypted on a machine with different endianness, now or in the future?
@user541686 2017-04-29 08:48:55
@KrisVandermotten: "What if it is to be decrypted on a machine with different endianness, now or in the future?" ...... sigh. How much of this answer & follow-up discussion did you read before posting your comment?? Literally the 2nd-most-upvoted comment -- which is the top comment before you expand the comments -- said the exact thing about endianness as you just did, and literally the 5th-most-upvoted comment -- which is the second comment before expansion -- was my reply to it... and they were from 5 years ago!!
@Kris Vandermotten 2017-04-29 09:09:51
@Mehrdad My second point is that your comment that "the whole point of this is if you want to use it on the same kind of system, with the same set of functions" is meaningless. Encryption is only useful if you do IO, write the encrypted stream to somewhere else, to be read back in a different place or time. You did not address that. More importantly it is trumped by my first point: why would anyone want to use your function? Is it less efficient than the built-in one.
@Kris Vandermotten 2017-04-29 09:11:48
And finally, if the use case was different, and OP really wanted to get to the bytes, then going unsafe and casting the
char* to a
byte* would be the most direct answer, not copying the string twice.
@user541686 2017-04-29 10:28:58
@KrisVandermotten: (a) This is a thing, (b) Unsafe code requires extra runtime privileges you might not have, (c) If someone writes or uses unsafe code wrong it'll silently corrupt memory instead of crashing, (d) Nowhere did I claim this is the fastest answer, (e) Nowhere did the OP claim he wants the fastest answer either, (f) Someone already posted an answer using unsafe code so go upvote it instead of arguing with me, (g) I'm just answering the question; if you don't like the OP's use case, go argue with him.
@emery.noel 2017-05-04 16:03:56
Hi @Mehrdad, I have 2 questions for you. In your first 2 paragraphs, you use the terms "interpreted" and "re-construct". What do you mean by "interpreted"? Can you provide an example of what interpretation means? "Re-construct" makes sense to me (bytes-to-readable string), but don't you have to use the same kind of encoding when you're re-constructing? I understand that on the same system it should not be a problem, but is it a potential problem between different systems? Thanks so much for your answers!
@user541686 2017-05-04 16:58:32
@emery.noel: I mean, do you care what the bytes are? Would anything in your code break if the bytes weren't valid UTF-16 (or UTF-8, or anything else your code is aware of)? If yes then you're interpreting them. As for different systems, yes it's a potential problem, and this answer is not for that scenario.
@Gerard ONeill 2017-11-01 19:18:11
I've changed my mind about this. Something I didn't see on original viewing -- Characters are a fixed size in c# -- this is really just an array copy. Creating the array might require interpretation; loading the array back to a string might also. But the array itself is recreated without interpretation because the Chars are the same size, which allows recreation of the original char array. That's all this depends on.
@mg30rg 2017-12-05 16:19:12
Are you aware of the fact that
length * sizeof(char) won't give you the size of the text in bytes? There are encodings like UTF-8 where the size of a character can vary. In case of UTF-8 it can be anything from 1 byte to four.
@John Rasch 2018-01-11 16:36:06
@chris's comments of "I use this solution for converting password strings to byte[] before salting and hashing them. In this use case, I absolutely do not care about encoding at all" should finally convince you to delete this answer. It is obviously not clear enough to be useful if someone actually believes that enough to defend it.
@chris 2018-01-13 17:34:23
@John Rasch I still don't see what's wrong with that. .NET strings always have the same, fixed-length encoding (i.e. UTF-16). So it's safe to assume that two .NET strings with identical char sequence are internally represented as identical byte sequence.
@chris 2018-01-13 17:49:13
I withdraw the "fixed-length" part of the comment above, that is admittedly incorrect. Still, I don't see why any two equal .NET strings should ever be represented as different byte sequences in memory.
@user541686 2018-01-28 04:13:03
string.Equals("\u0041\u030A", "\u00C5", StringComparison.InvariantCulture) is one example, but it also has absolutely nothing to do with my answer, since you would have exactly the same problem if you specify an encoding.
@dodexahedron 2018-01-29 09:49:00
The fact that the original data is stored in a string already implies encoding. It's not merely an array of bytes to be toyed with as you please. If it were so, why have you stored it in a string? That's...Just stupid. The assertion being made in here that people are incorrectly "interpreting" the bytes is flat-out incorrect, because they bytes have already been interpreted by the fact the original data was stored in a .net string. The consumer of the resulting bytes is going to have to implicitly know what the encoding was to make any use of the original bytes whatsoever.
@Tom Lint 2018-02-15 09:04:10
This answer is so wrong, I'm shocked to see it's got this many upvotes. Yes, in theory it works. But that's exactly where the possible use cases for this code end. Anyone using this code in production should be fired on the spot. And the argument "It doesn't matter if the string contains invalid characters" is BS, because your strings will never contain invalid characters to begin with.
@user253751 2018-07-10 22:25:09
This is an encoding. You've just invented your own encoding instead of using a standard one.
@Chris F Carroll 2018-10-10 16:22:33
Surely your bolded extra-large headline should be "you DON'T need to worry about encoding when the platform handles it for you".
@DrewB 2019-01-10 16:58:36
The intent the OP stated was encrypting strings. I would guess if a string were being encrypted then it would also be transported at some point. What would happen, using this method, if one system were using UTF-8 and the other were using UTF-16? Wouldn't that throw off sizeof(char) and completely mangle the string?
@user541686 2019-01-10 17:45:08
@DrewB: yes, because then you'd have written buggy code. This code is doing its job just fine, and it does no more.
@Matt Thomas 2019-06-03 20:03:12
But moving along, since you did mean "injection" then I think your answer could be improved by making clear (in the answer text and not just comments) that
GetString does not work for arbitrary byte arrays but only for those produced by
GetBytes. I think this also implies either (a) removing the statement "you DON'T need to worry about encoding if the bytes don't need to be interpreted" because the use of
GetString implies interpretation (since only "meaningful" byte arrays work as input) or (b) removing
GetString itself (which technically isn't necessary for OP). Thoughts?
@user541686 2019-06-03 20:08:10
@MattThomas: I can't remove your comments since I'm not a mod. Removing
GetString would make
GetBytes pointless since you need to be able to reconstruct the original string. I never suggested
GetBytes works on "any string" either, that's something else that you read into my comments out of nowhere. If you simply try it it will clearly fail half the time. I was explaining how to get bytes from a string. Re: the note, I didn't add it in the first place; people argued & nitpicked & flamed me for >3 years until someone else added it. I'm sick of arguing over random nitpicking at this point.
@Matt Thomas 2019-06-03 20:13:03
"I'm sick of arguing over random nitpicking at this point" ROFL! I'm sorry for laying on more of it, just trying to be clear and precise in short space. I'm not saying that you said it does work on arbitrary byte arrays, just that the answer could be improved in a small way so that people don't falsely assume it does when they cargo code. I'll make the edit and we'll see how the mobs respond :)
@verdy_p 2019-09-07 16:26:02
beware that
str.ToCharArray() actually uses an hidden "encoding" to convert the 16-bit codeunits to arrays of Char; if it's not lossy, it will push at least two chars per codeunit in the string; if it was lossy, it will drop the highbyte of each code unit to return one Char. Then this depends on the bitsize of each "Char" (which is not necessarily an 8-bit "byte"). Look for the difference between "byte" and "char" data types, they're not equivalent in C# (same thing in C and C++). To allow lossless conversions with strings without exceptions, a "char" needs to be an unsigned 16-bit exactly.
@verdy_p 2019-09-07 16:31:35
For being lossless, a 16-bit "char" will allow arbitrary codeunits in any order (including unpaired surrogates). So an array of "char" it is also not warrantied to be valid UTF-16. Strings are just convenient compacted and unmutable arrays of 16-bit units, frequently (not always) more efficient in storage and processing speed compared to classic mutable arrays of numbers.
@jpmc26 2019-09-11 03:26:52
Because the .NET runtime might change its encoding in some future version. Then your code will break if the runtimes of two systems need to communicate are different or if the runtime is upgraded after storing the bytes obtained in this manner. I'm not sure the internal representation of a string is even guaranteed or documented. An explicit encoding will be far less likely to suffer from these problems.
@Eike 2019-10-22 12:06:53
It is impossible to convert a sequence of letters to an array of numbers (char in this case) without at least implicitly using an encoding. Which is why MSDN describes ToCharArray as "Copies the characters in this instance to a Unicode character array." and char as "a character as a UTF-16 code unit."
@joe 2019-11-27 11:21:59
You just assume the input is standard English string. How can you make sure
str.Length * sizeof(char) is the size of the string? I mean the string doesn't even have to be English characters. For example, in UTF-8 "你好", which returns length as 2, but each of them is 3 bytes instead of 1 byte in UTF-8!
@WonderWorker 2014-04-09 12:39:54
To convert a string to a byte[] use the following solution:
string s = "abcdefghijklmnopqrstuvwxyz";
byte[] b = System.Text.UTF32Encoding.GetBytes(s);
I hope it helps.
@Sebastian 2014-04-12 17:12:59
that's not a solution!
@WonderWorker 2014-04-13 03:20:58
Why would you say that?
@Sebastian 2014-04-13 08:04:29
Before your edit it was:
s.Select(e => (byte)e) this only works for ASCII characters. But the
char type is for storing UTF16 Units. Now after your editing, the code is at least correct, but it varies from environment to environment, hence rendering it virtually useless. IMHO Encoding.Default should only be used for interacting with legacy Windows "Ansi codepage" code.
@WonderWorker 2014-04-14 08:30:58
Good point. How do you feel about byte[] b = new System.Text.UTF32Encoding().GetBytes(s); ?
@Sebastian 2014-04-14 09:12:38
byte[] b = System.Text.UTF32Encoding.GetBytes(s);, UTF8 is equally fine.
@Jason Goemaat 2018-07-02 20:51:49
How do I convert a string to a byte[] in .NET (C#) without manually specifying a specific encoding?
A string in .NET represents text as a sequence of UTF-16 code units, so the bytes are encoded in memory in UTF-16 already.
Mehrdad's Answer
You can use Mehrdad's answer, but it does actually use an encoding because chars are UTF-16. It calls ToCharArray which looking at the source creates a
char[] and copies the memory to it directly. Then it copies the data to a byte array that is also allocated. So under the hood it is copying the underlying bytes twice and allocating a char array that is not used after the call.
Tom Blodget's Answer
Tom Blodget's answer is 20-30% faster than Mehrdad since it skips the intermediate step of allocating a char array and copying the bytes to it, but it requires you compile with the
/unsafe option. If you absolutely do not want to use encoding, I think this is the way to go. If you put your encryption login inside the
fixed block, you don't even need to allocate a separate byte array and copy the bytes to it.
Also, why should encoding be taken into consideration? Can't I simply get what bytes the string has been stored in? Why is there a dependency on character encodings?
Because that is the proper way to do it.
string is an abstraction.
Using an encoding could give you trouble if you have 'strings' with invalid characters, but that shouldn't happen. If you are getting data into your string with invalid characters you are doing it wrong. You should probably be using a byte array or a Base64 encoding to start with.
If you use
System.Text.Encoding.Unicode, your code will be more resilient. You don't have to worry about the endianness of the system your code will be running on. You don't need to worry if the next version of the CLR will use a different internal character encoding.
I think the question isn't why you want to worry about the encoding, but why you want to ignore it and use something else. Encoding is meant to represent the abstraction of a string in a sequence of bytes.
System.Text.Encoding.Unicode will give you a little endian byte order encoding and will perform the same on every system, now and in the future.
@verdy_p 2019-09-07 15:42:45
Actually a string in C# is NOT restricted to just UTF-16. What is true is that it contains a vector of 16-bit code units, but these 16-bit code units are not restricted to valid UTF-16. But as they are 16-bit, you need an encoding (byte order) to convert them to 8bit. A string can then store non-Unicode data, including binary code (e.g. a bitmap image). It becomes interpreted as UTF-16 only in I/O and text formatters that make such interpretation.
@verdy_p 2019-09-07 15:47:52
So in a C# string, you can safely store a code unit like 0xFFFF or 0xFFFE, even if they are non-characters in UTF-16, and you can store an isolated 0xD800 not followed by a code unit in 0xDC00..0xDFFF (i.e. unpaired surrogates which are invalid in UTF-16). The same remark applies to strings in Javascript/ECMAscript and Java.
@verdy_p 2019-09-07 15:55:58
When you use "GetBytes", of course you don't specify an encoding, but you assume a byte order to get the two bytes in a specic for each code unit stored locally in the string. When you build a new string from bytes, you also need a converter, not necessarily UTF-8 to UTF-16, you could insert the extra 0 in the high byte, or pack two bytes (in MSB first or LSB first order) in the same 16-bit code unit. Strings are then compact form for arrays of 16-bit integers. The relation with "characters" is another problem, in C# they're not actual types as they are still represented as strings
@John Rasch 2018-01-10 20:21:12
With the advent of
Span<T> released with C# 7.2, the canonical technique to capture the underlying memory representation of a string into a managed byte array is:
byte[] bytes = "rubbish_\u9999_string".AsSpan().AsBytes().ToArray();
Converting it back should be a non-starter because that means you are in fact interpreting the data somehow, but for the sake of completeness:
string s;
fixed (char* f = &bytes.AsSpan().NonPortableCast<byte, char>().DangerousGetPinnableReference())
s = new string(f);
The names
NonPortableCast and
DangerousGetPinnableReference should further the argument that you probably shouldn't be doing this.
Note that working with
Span<T> requires installing the System.Memory NuGet package.
Regardless, the actual original question and follow-up comments imply that the underlying memory is not being "interpreted" (which I assume means is not modified or read beyond the need to write it as-is), indicating that some implementation of the
Stream class should be used instead of reasoning about the data as strings at all.
It depends on what you want the bytes FOR
This is because, as Tyler so aptly said, "Strings aren't pure data. They also have information." In this case, the information is an encoding that was assumed when the string was created.
Assuming that you have binary data (rather than text) stored in a string
This is based off of OP's comment on his own question, and is the correct question if I understand OP's hints at the use-case.
Storing binary data in strings is probably the wrong approach because of the assumed encoding mentioned above! Whatever program or library stored that binary data in a
string (instead of a
byte[] array which would have been more appropriate) has already lost the battle before it has begun. If they are sending the bytes to you in a REST request/response or anything that must transmit strings, Base64 would be the right approach.
If you have a text string with an unknown encoding
Everybody else answered this incorrect question incorrectly.
If the string looks good as-is, just pick an encoding (preferably one starting with UTF), use the corresponding
System.Text.Encoding.???.GetBytes() function, and tell whoever you give the bytes to which encoding you picked.
@Gerard ONeill 2015-08-18 17:04:21
The closest approach to the OP's question is Tom Blodget's, which actually goes into the object and extracts the bytes. I say closest because it depends on implementation of the String Object.
"Can't I simply get what bytes the string has been stored in?"
Sure, but that's where the fundamental error in the question arises. The String is an object which could have an interesting data structure. We already know it does, because it allows unpaired surrogates to be stored. It might store the length. It might keep a pointer to each of the 'paired' surrogates allowing quick counting. Etc. All of these extra bytes are not part of the character data.
What you want is each character's bytes in an array. And that is where 'encoding' comes in. By default you will get UTF-16LE. If you don't care about the bytes themselves except for the round trip then you can choose any encoding including the 'default', and convert it back later (assuming the same parameters such as what the default encoding was, code points, bug fixes, things allowed such as unpaired surrogates, etc.
But why leave the 'encoding' up to magic? Why not specify the encoding so that you know what bytes you are gonna get?
"Why is there a dependency on character encodings?"
Encoding (in this context) simply means the bytes that represent your string. Not the bytes of the string object. You wanted the bytes the string has been stored in -- this is where the question was asked naively. You wanted the bytes of string in a contiguous array that represent the string, and not all of the other binary data that a string object may contain.
Which means how a string is stored is irrelevant. You want a string "Encoded" into bytes in a byte array.
I like Tom Bloget's answer because he took you towards the 'bytes of the string object' direction. It's implementation dependent though, and because he's peeking at internals it might be difficult to reconstitute a copy of the string.
Mehrdad's response is wrong because it is misleading at the conceptual level. You still have a list of bytes, encoded. His particular solution allows for unpaired surrogates to be preserved -- this is implementation dependent. His particular solution would not produce the string's bytes accurately if
GetBytes returned the string in UTF-8 by default.
I've changed my mind about this (Mehrdad's solution) -- this isn't getting the bytes of the string; rather it is getting the bytes of the character array that was created from the string. Regardless of encoding, the char datatype in c# is a fixed size. This allows a consistent length byte array to be produced, and it allows the character array to be reproduced based on the size of the byte array. So if the encoding were UTF-8, but each char was 6 bytes to accommodate the largest utf8 value, it would still work. So indeed -- encoding of the character does not matter.
But a conversion was used -- each character was placed into a fixed size box (c#'s character type). However what that representation is does not matter, which is technically the answer to the OP. So -- if you are going to convert anyway... Why not 'encode'?
@Mojtaba Rezaeian 2016-02-11 19:48:49
These characters are not supported by UTF-8 or UTF-16 or even UTF-32 for exapmle:
(Char) 55906 &
(Char) 55655. So you may be wrong and Mehrdad's answer is a safe conversion without considering what type of encodings are used.
@Gerard ONeill 2016-02-11 20:47:14
Raymon, the characters are already represented by some unicode value -- and all unicode values can be represented by all the utf's. Is there a longer explanation of what you are talking about? What character encoding do those two values (or 3..) exist in?
@Mojtaba Rezaeian 2016-02-11 21:02:37
They are invalid characters which not supported by any encoding ranges. This not means they are 100% useless. A code which converts any type of string to its byte array equivalent regardless of the encodings is not a wrong solution at all and have its own usages on desired occasions.
@Gerard ONeill 2016-02-11 22:17:24
Ok, then I think you are not understanding the problem. We know it is a unicode compliant array -- in fact, because it is .net, we know it is UTF-16. So those characters will not exist there. You also didn't fully read my comment about internal representations changing. A String is an object, not an encoded byte array. So I'm going to disagree with your last statement. You want code to convert all unicode strings to any UTF encoding. This does what you want, correctly.
@Mojtaba Rezaeian 2016-02-11 23:00:41
Objects are sequence of data originally sequence of bits which describe an object in its current state. So every data in programming languages are convertible to array of bytes(each byte defines 8 bits) as you may need to keep some state of any object in memory. You can save and hold a sequence of bytes in file or memory and cast it as integer, bigint, image, Ascii string, UTF-8 string, encrypted string, or your own defined datatype after reading it from disk. So you can not say objects are something different than bytes sequence.
@Gerard ONeill 2017-11-01 19:49:52
Mojtaba -- I updated my answer with a wiser mind at the keyboard. However what you said isn't right for objects who have other object dependencies. But Mehrdad's solution, by converting it to an array of char, eliminates this, making what you said possible. Still trying to decide whether or not to replace my entire response.. But perhaps my learning process will have some value.
@Joel Coehoorn 2009-01-23 15:54:28
The first part of your question (how to get the bytes) was already answered by others: look in the
System.Text.Encoding namespace.
I will address your follow-up question: why do you need to pick an encoding? Why can't you get that from the string class itself?
The answer is in two parts.
First of all, the bytes used internally by the string class don't matter, and whenever you assume they do you're likely introducing a bug.
If your program is entirely within the .Net world then you don't need to worry about getting byte arrays for strings at all, even if you're sending data across a network. Instead, use .Net Serialization to worry about transmitting the data. You don't worry about the actual bytes any more: the Serialization formatter does it for you.
On the other hand, what if you are sending these bytes somewhere that you can't guarantee will pull in data from a .Net serialized stream? In this case you definitely do need to worry about encoding, because obviously this external system cares. So again, the internal bytes used by the string don't matter: you need to pick an encoding so you can be explicit about this encoding on the receiving end, even if it's the same encoding used internally by .Net.
I understand that in this case you might prefer to use the actual bytes stored by the string variable in memory where possible, with the idea that it might save some work creating your byte stream. However, I put it to you it's just not important compared to making sure that your output is understood at the other end, and to guarantee that you must be explicit with your encoding. Additionally, if you really want to match your internal bytes, you can already just choose the
Unicode encoding, and get that performance savings.
Which brings me to the second part... picking the
Unicode encoding is telling .Net to use the underlying bytes. You do need to pick this encoding, because when some new-fangled Unicode-Plus comes out the .Net runtime needs to be free to use this newer, better encoding model without breaking your program. But, for the moment (and forseeable future), just choosing the Unicode encoding gives you what you want.
It's also important to understand your string has to be re-written to wire, and that involves at least some translation of the bit-pattern even when you use a matching encoding. The computer needs to account for things like Big vs Little Endian, network byte order, packetization, session information, etc.
@Ash 2010-01-28 09:33:21
There are areas in .NET where you do have to get byte arrays for strings. Many of the .NET Cryptrography classes contain methods such as ComputeHash() that accept byte array or stream. You have no alternative but to convert a string to a byte array first (choosing an Encoding) and then optionally wrap it in a stream. However as long as you choose an encoding (ie UTF8) an stick with it there are no problems with this.
@Vijay Singh Rana 2014-06-11 11:29:06
The string can be converted to byte array in few different ways, due to the following fact: .NET supports Unicode, and Unicode standardizes several difference encodings called UTFs. They have different lengths of byte representation but are equivalent in that sense that when a string is encoded, it can be coded back to the string, but if the string is encoded with one UTF and decoded in the assumption of different UTF if can be screwed up.
Also, .NET supports non-Unicode encodings, but they are not valid in general case (will be valid only if a limited sub-set of Unicode code point is used in an actual string, such as ASCII). Internally, .NET supports UTF-16, but for stream representation, UTF-8 is usually used. It is also a standard-de-facto for Internet.
Not surprisingly, serialization of string into an array of byte and deserialization is supported by the class
System.Text.Encoding, which is an abstract class; its derived classes support concrete encodings:
ASCIIEncoding and four UTFs (
System.Text.UnicodeEncoding supports UTF-16)
Ref this link.
For serialization to an array of bytes using
System.Text.Encoding.GetBytes. For the inverse operation use
System.Text.Encoding.GetChars. This function returns an array of characters, so to get a string, use a string constructor
Ref this page.
string myString = //... some string
System.Text.Encoding encoding = System.Text.Encoding.UTF8; //or some other, but prefer some UTF is Unicode is used
byte[] bytes = encoding.GetBytes(myString);
//next lines are written in response to a follow-up questions:
myString = new string(encoding.GetChars(bytes));
byte[] bytes = encoding.GetBytes(myString);
myString = new string(encoding.GetChars(bytes));
byte[] bytes = encoding.GetBytes(myString);
//how many times shall I repeat it to show there is a round-trip? :-)
@Gman 2011-03-10 08:57:30
Well, I've read all answers and they were about using encoding or one about serialization that drops unpaired surrogates.
It's bad when the string, for example, comes from SQL Server where it was built from a byte array storing, for example, a password hash. If we drop anything from it, it'll store an invalid hash, and if we want to store it in XML, we want to leave it intact (because the XML writer drops an exception on any unpaired surrogate it finds).
So I use Base64 encoding of byte arrays in such cases, but hey, on the Internet there is only one solution to this in C#, and it has bug in it and is only one way, so I've fixed the bug and written back procedure. Here you are, future googlers:
public static byte[] StringToBytes(string str)
byte[] data = new byte[str.Length * 2];
for (int i = 0; i < str.Length; ++i)
char ch = str[i];
data[i * 2] = (byte)(ch & 0xFF);
data[i * 2 + 1] = (byte)((ch & 0xFF00) >> 8);
return data;
public static string StringFromBytes(byte[] arr)
char[] ch = new char[arr.Length / 2];
for (int i = 0; i < ch.Length; ++i)
ch[i] = (char)((int)arr[i * 2] + (((int)arr[i * 2 + 1]) << 8));
return new String(ch);
@Makotosan 2012-02-10 15:53:36
Instead of using your custom method to convert a byte array to base64, all you had to do was use the built-in converter: Convert.ToBase64String(arr);
@Gman 2012-03-06 19:15:02
@Makotosan thank you, but I did use
Convert.ToBase64String(arr); for the base64 conversions
byte[] (data) <-> string (serialized data to store in XML file). But to get the initial
byte[] (data) I needed to do something with a
String that contained binary data (it's the way MSSQL returned it to me). SO the functions above are for
String (binary data) <-> byte[] (easy accessible binary data).
@mashet 2013-10-22 12:55:59
string text = "string";
byte[] array = System.Text.Encoding.UTF8.GetBytes(text);
The result is:
[0] = 115
[1] = 116
[2] = 114
[3] = 105
[4] = 110
[5] = 103
@Ferdz 2018-08-30 13:40:21
OP specifically asks to NOT specify an encoding... "without manually specifying a specific encoding"
@George 2014-08-28 16:14:16
A character is both a lookup key into a font table and a lexical tradition such as ordering, upper and lower case versions, etc.
Consequently, a character is not a byte (8-bits) and a byte is not a character. In particular, the 256 permutations of a byte cannot accommodate the thousands of symbols within some written languages, much less all languages. Hence, various methods for encoding characters have been devised. Some encode for a particular class of languages (ASCII encoding); multiple languages using code pages (Extended ASCII); or, ambitiously, all languages by selectively including additional bytes as needed, Unicode.
Within a system, such as the .NET framework, a String implies a particular character encoding. In .NET this encoding is Unicode. Since the framework reads and writes Unicode by default, dealing with character encoding is typically not necessary in .NET.
However, in general, to load a character string into the system from a byte stream you need to know the source encoding to therefore interpret and subsequently translate it correctly (otherwise the codes will be taken as already being in the system's default encoding and thus render gibberish). Similarly, when a string is written to an external source, it will be written in a particular encoding.
@Kevin 2017-08-26 03:22:50
Unicode is not an encoding. Unicode is an abstract mapping of characters to codepoints. There are multiple ways of encoding Unicode; in particular, UTF-8 and UTF-16 are most common. .NET uses UTF-16, though I'm unsure if it's UTF-16 LE or UTF-16 BE.
@verdy_p 2019-09-07 16:05:49
UTF-16 LE or UTF-16 BE is nor relevant: strings are using unbreakable 16-bit code units without any interpretation. UTF-16BE or UTF-16 LE may become relevant only when you convert strings to byte arrays or the reverse because, at that time, you'll specify an encoding (and in that case the string must first be valid UTF-16, but strings don't have to be valid UTF-16). GetBytes() is not necessarily returning valid UTF-16 BE/LE, it uses a simple arithmetic; the returned array is also not valid UTF-8 but arbitrary bytes. The byte order in result is system-specific if no encoding is specified.
@verdy_p 2019-09-07 16:15:53
This also means that string.UTF8.getBytes() may throw encoding exceptions from arbitrary strings whose content is not valid UTF-16. In C# you have the choice of encoders/decoders (codec) to use. You may want to use your own codec which will pack/unpack bytes differently, or may silently drop unpaired surrogates (if the codec attempts to interpret the string as UTF-16), or may drop the high bytes, or replace/interpret the codeunits invalid in UTF-16 by U+FFFD. The codec may also use data compression, or hexadecimal/base64 or escaping...Codecs are not restricted to just the UTF8 encoding.
@verdy_p 2019-09-07 18:50:54
note: I use here the term "codec" voluntarily instead of "encoding" which is more specific and used only for text. strings in C#, C, C++, Java, Javascript/ECMAscript/ActiveScript are NOT restricted to just valid text: they are just a generic storage structure, convenient for text and treated as text by libraries (but not all). As such the UTF forms are not enforced at all except inside specific APIs using them (including UTF* encoding objects). Yes you can store a binary program or PNG image in a compact immutable string instead of mutable array, but you can I/O all strings to text channels
@Piero Alberto 2015-01-21 14:05:34
byte[] to
return BitConverter.ToString(bytes);
@Mojtaba Rezaeian 2016-02-11 19:32:12
I have written a Visual Basic extension similar to the accepted answer, but directly using .NET memory and Marshalling for conversion, and it supports character ranges unsupported in other methods, like
UnicodeEncoding.UTF8.GetString or
UnicodeEncoding.UTF32.GetString or even
MemoryStream and BinaryFormatter (invalid characters like:
ChrW(55906) &
<Extension> _
Public Function ToBytesMarshal(ByRef str As String) As Byte()
Dim gch As GCHandle = GCHandle.Alloc(str, GCHandleType.Pinned)
Dim handle As IntPtr = gch.AddrOfPinnedObject
ToBytesMarshal = New Byte(str.Length * 2 - 1) {}
For i As Integer = 0 To ToBytesMarshal.Length - 1
ToBytesMarshal.SetValue(Marshal.ReadByte(IntPtr.Add(handle, i)), i)
End Try
End Function
<Extension> _
Public Function ToStringMarshal(ByRef arr As Byte()) As String
Dim gch As GCHandle = GCHandle.Alloc(arr, GCHandleType.Pinned)
ToStringMarshal = Marshal.PtrToStringAuto(gch.AddrOfPinnedObject)
End Try
End Function
@Shyam sundar shah 2013-06-05 10:52:33
C# to convert a
string to a
byte array:
public static byte[] StrToByteArray(string str)
System.Text.UTF8Encoding encoding=new System.Text.UTF8Encoding();
return encoding.GetBytes(str);
@Ed Marty 2009-01-23 14:34:03
I'm not sure, but I think the string stores its info as an array of Chars, which is inefficient with bytes. Specifically, the definition of a Char is "Represents a Unicode character".
take this example sample:
String str = "asdf éß";
String str2 = "asdf gh";
EncodingInfo[] info = Encoding.GetEncodings();
foreach (EncodingInfo enc in info)
System.Console.WriteLine(enc.Name + " - "
+ enc.GetEncoding().GetByteCount(str)
+ enc.GetEncoding().GetByteCount(str2));
Take note that the Unicode answer is 14 bytes in both instances, whereas the UTF-8 answer is only 9 bytes for the first, and only 7 for the second.
So if you just want the bytes used by the string, simply use
Encoding.Unicode, but it will be inefficient with storage space.
@Tommaso Belluzzo 2013-01-15 11:43:42
Here is my unsafe implementation of
String to
Byte[] conversion:
public static unsafe Byte[] GetBytes(String s)
Int32 length = s.Length * sizeof(Char);
Byte[] bytes = new Byte[length];
fixed (Char* pInput = s)
fixed (Byte* pBytes = bytes)
Byte* source = (Byte*)pInput;
Byte* destination = pBytes;
if (length >= 16)
*((Int64*)destination) = *((Int64*)source);
*((Int64*)(destination + 8)) = *((Int64*)(source + 8));
source += 16;
destination += 16;
while ((length -= 16) >= 16);
if (length > 0)
if ((length & 8) != 0)
*((Int64*)destination) = *((Int64*)source);
source += 8;
destination += 8;
if ((length & 4) != 0)
*((Int32*)destination) = *((Int32*)source);
source += 4;
destination += 4;
if ((length & 2) != 0)
*((Int16*)destination) = *((Int16*)source);
source += 2;
destination += 2;
if ((length & 1) != 0)
destination[0] = source[0];
return bytes;
It's way faster than the accepted anwser's one, even if not as elegant as it is. Here are my Stopwatch benchmarks over 10000000 iterations:
[Second String: Length 20]
Buffer.BlockCopy: 746ms
Unsafe: 557ms
[Second String: Length 50]
Buffer.BlockCopy: 861ms
Unsafe: 753ms
[Third String: Length 100]
Buffer.BlockCopy: 1250ms
Unsafe: 1063ms
In order to use it, you have to tick "Allow Unsafe Code" in your project build properties. As per .NET Framework 3.5, this method can also be used as String extension:
public static unsafe class StringExtensions
public static Byte[] ToByteArray(this String s)
// Method Code
@Jon Hanna 2014-01-06 14:09:00
Is the value of
RuntimeHelpers.OffsetToStringData a multiple of 8 on the Itanium versions of .NET? Because otherwise this will fail due to the unaligned reads.
@Jodrell 2014-11-25 10:33:45
wouldn't it be simpler to invoke
memcpy? stackoverflow.com/a/27124232/659190
@Alessandro Annini 2010-03-22 08:40:52
Fastest way
public static byte[] GetBytes(string text)
return System.Text.ASCIIEncoding.UTF8.GetBytes(text);
EDIT as Makotosan commented this is now the best way:
@Makotosan 2012-02-17 20:40:25
ASCIIEncoding..... is not needed. Simply using Encoding.UTF8.GetBytes(text) is preferred.
@IgnusFast 2016-01-21 17:19:03
I had to convert a string to a byte array for a serial communication project - I had to handle 8-bit characters, and I was unable to find a method using the framework converters to do so that didn't either add two-byte entries or mis-translate the bytes with the eighth bit set. So I did the following, which works:
string message = "This is a message.";
byte[] bytes = new byte[message.Length];
for (int i = 0; i < message.Length; i++)
bytes[i] = (byte)message[i];
@Mojtaba Rezaeian 2016-02-11 19:43:09
Its not safe this way and you will loose original data if input string contains unicode range characters.
@IgnusFast 2017-02-06 20:55:04
This was for a serial communication project, which couldn't handle unicode anyway. Granted that it was an extremely narrow case.
@Konamiman 2009-07-16 11:45:14
Also please explain why encoding should be taken into consideration. Can't I simply get what bytes the string has been stored in? Why this dependency on encoding?!!!
Because there is no such thing as "the bytes of the string".
A string (or more generically, a text) is composed of characters: letters, digits, and other symbols. That's all. Computers, however, do not know anything about characters; they can only handle bytes. Therefore, if you want to store or transmit text by using a computer, you need to transform the characters to bytes. How do you do that? Here's where encodings come to the scene.
An encoding is nothing but a convention to translate logical characters to physical bytes. The simplest and best known encoding is ASCII, and it is all you need if you write in English. For other languages you will need more complete encodings, being any of the Unicode flavours the safest choice nowadays.
So, in short, trying to "get the bytes of a string without using encodings" is as impossible as "writing a text without using any language".
By the way, I strongly recommend you (and anyone, for that matter) to read this small piece of wisdom: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
@Agnel Kurian 2009-07-16 15:30:48
Allow me to clarify: An encoding has been used to translate "hello world" to physical bytes. Since the string is stored on my computer, I am sure that it must be stored in bytes. I merely want to access those bytes to save them on disk or for any other reason. I do not want to interpret these bytes. Since I do not want to interpret these bytes, the need for an encoding at this point is as misplaced as requiring a phone line to call printf.
@Konamiman 2009-07-22 08:35:39
But again, there is no concept of text-to-physical-bytes-translation unless yo use an encoding. Sure, the compiler stores the strings somehow in memory - but it is just using an internal encoding, which you (or anyone except the compiler developer) do not know. So, whatever you do, you need an encoding to get physical bytes from a string.
@ollb 2011-05-14 00:06:56
@Agnel Kurian: It is of course true, that a string has a bunch of bytes somewhere that store its content (UTF-16 afair). But there is a good reason to prevent you from accessing it: strings are immutable and if you could obtain the internal byte[] array, you could modify it, too. This breaks immutability, which is vital because multiple strings may share the same data. Using an UTF-16 encoding to get the string will probably just copy the data out.
@Agnel Kurian 2011-05-14 05:06:15
@Gnafoo, A copy of the bytes will do.
@Erik A. Brandstadmoen 2012-04-30 07:26:07
The accepted answer is very, very complicated. Use the included .NET classes for this:
const string data = "A string with international characters: Norwegian: ÆØÅæøå, Chinese: 喂 谢谢";
var bytes = System.Text.Encoding.UTF8.GetBytes(data);
var decoded = System.Text.Encoding.UTF8.GetString(bytes);
Don't reinvent the wheel if you don't have to...
@Thomas Eding 2013-09-27 18:20:56
In case the accepted answer gets changed, for record purposes, it is Mehrdad's answer at this current time and date. Hopefully the OP will revisit this and accept a better solution.
@Jodrell 2014-11-25 09:08:45
good in principle but, the encoding should be
System.Text.Encoding.Unicode to be equivalent to Mehrdad's answer.
@Erik A. Brandstadmoen 2014-11-26 11:36:07
The question has been edited an umptillion times since the original answer, so, maybe my answer is a bit outdates. I never intended to give an exace equivalent to Mehrdad's answer, but give a sensible way of doing it. But, you might be right. However, the phrase "get what bytes the string has been stored in" in the original question is very unprecise. Stored, where? In memory? On disk? If in memory,
System.Text.Encoding.Unicode.GetBytes would probably be more precise.
@AMissico 2016-02-10 21:37:46
After reviewing all the answers, the many comments, and my inspection of memory (don't forget, Visual Studio allows for memory inspection) that the correct answer is
@Frédéric 2016-04-06 20:53:26
@AMissico, your suggestion is buggy, unless you are sure your string is compatible with your system default encoding (string containing only ASCII chars in your system default legacy charset). But nowhere the OP states that.
@AMissico 2016-04-07 18:24:03
@Frédéric; I am just stating my opinion after reviewing all the information and running test scenarios with Unicode characters. I have also used TextPad, HexEdit, WinHex, and Visual Studio to view those bytes. The
Encoding.Default.GetBytes results are the same as those applications. I am not providing an answer to the OP question.
@Nyerguds 2016-04-22 10:33:15
@AMissico It can cause the program to give different results on different systems though. That's never a good thing. Even if it's for making a hash or something (I assume that's what OP means with 'encrypt'), the same string should still always give the same hash.
@jinzai 2016-06-22 15:11:02
+1 for UTF-8. That is what is being assumed by those that say encoding does not matter. UTF-8 is a strict value for value encoding of an unsigned char (BYTE). Everything else is...not.
@jinzai, but what about UTF-16, which .NET uses internally?
@jinzai 2017-11-10 15:13:55
UTF-16 is part of the "everything else" I mentioned. The original question -- was referring to 'byte representations'. With respect to UTF-16 -- the values map the same for ASCII, but -- they are words, not bytes. I am fairly certain that everyone knows that .NET uses UTF-16 internally, however -- I always use UTF-8 for things like XML. .NET now respects that, at least.
@Nyerguds 2019-12-17 13:32:47
@NH. Does it, though? From what I've seen it stores text as raw 16 bit unicode values, not as UTF-16. The UTF-16 encoding includes control bits and is expandable, but .Net simply only supports unicode symbols from 0x0000 to 0xFFFF.
@Nyerguds, I'm just referring to the first sentence in the manpage for String
@Nyerguds 2020-01-10 17:22:58
@NH. MS has a bad habit of constantly mixing up 'Unicode' and 'UTF-16'. Case in point, their UTF-16 encoding class is called
Unicode. Same for strings; click through to the
Char page, and you get this gem: "The Unicode Standard identifies each Unicode character with a unique 21-bit scalar number called a code point, and defines the UTF-16 encoding form that specifies how a code point is encoded into a sequence of one or more 16-bit values." This is nonsense; UTF-16 encoding is one system that works on any given value, unrelated to what the unicode standards define that value as.
@alireza amini 2015-06-30 14:39:07
Simply use this:
byte[] myByte= System.Text.ASCIIEncoding.Default.GetBytes(myString);
@mg30rg 2018-01-11 15:09:12
...and lose all characters with a jump cope higher than 127. In my native language it is perfectly valid to write "Árvíztűrő tükörfúrógép.".
System.Text.ASCIIEncoding.Default.GetBytes("Árvíztűrő tükörfúrógép.").ToString(); will return
"Árvizturo tukörfurogép." losing information which can not be retrieved. (And I didn't yet mention asian languages where you would loose all characters.)
@Nathan 2011-07-25 22:52:28
Try this, a lot less code:
System.Text.Encoding.UTF8.GetBytes("TEST String");
@mg30rg 2017-12-05 16:30:19
Then try this
System.Text.Encoding.UTF8.GetBytes("Árvíztűrő tükörfúrógép);, and cry! It will work, but
System.Text.Encoding.UTF8.GetBytes("Árvíztűrő tükörfúrógép").Length != System.Text.Encoding.UTF8.GetBytes("Arvizturo tukorfurogep").Length while
"Árvíztűrő tükörfúrógép".Length == "Arvizturo tukorfurogep".Length
@Vlad 2018-02-25 01:18:35
@mg30rg: Why do you think your example is strange? Surely in a variable-width encoding not all characters have the same byte lengthes. What's wrong with it?
@Nyerguds 2020-03-31 12:43:54
@Vlad A more valid comment here, though, is that as encoded unicode symbols (so, as bytes), characters which include their own diacritics will give a different result than diacritics split off into modifier symbols added to the character. But iirc there are methods in .net to specifically split those off, to allow getting a consistent byte representation.
@bmotmans 2009-01-23 13:43:51
It depends on the encoding of your string (ASCII, UTF-8, ...).
For example:
byte[] b1 = System.Text.Encoding.UTF8.GetBytes (myString);
byte[] b2 = System.Text.Encoding.ASCII.GetBytes (myString);
A small sample why encoding matters:
string pi = "\u03a0";
byte[] ascii = System.Text.Encoding.ASCII.GetBytes (pi);
byte[] utf8 = System.Text.Encoding.UTF8.GetBytes (pi);
Console.WriteLine (ascii.Length); //Will print 1
Console.WriteLine (utf8.Length); //Will print 2
Console.WriteLine (System.Text.Encoding.ASCII.GetString (ascii)); //Will print '?'
ASCII simply isn't equipped to deal with special characters.
Internally, the .NET framework uses UTF-16 to represent strings, so if you simply want to get the exact bytes that .NET uses, use
System.Text.Encoding.Unicode.GetBytes (...).
See Character Encoding in the .NET Framework (MSDN) for more information.
@Agnel Kurian 2009-01-23 13:48:26
But, why should encoding be taken into consideration? Why can't I simply get the bytes without having to see what encoding is being used? Even if it were required, shouldn't the String object itself know what encoding is being used and simply dump what is in memory?
@AnthonyWJones 2009-01-23 14:33:29
A .NET strings are always encoded as Unicode. So use System.Text.Encoding.Unicode.GetBytes(); to get the set of bytes that .NET would using to represent the characters. However why would you want that? I recommend UTF-8 especially when most characters are in the western latin set.
@Joel Coehoorn 2009-01-23 15:39:08
There's also System.Text.Encoding.Default
@Joel Coehoorn 2009-01-23 15:42:16
Also: the exact bytes used internally in the string don't matter if the system that retrieves them doesn't handle that encoding or handles it as the wrong encoding. If it's all within .Net, why convert to an array of bytes at all. Otherwise, it's better to be explicit with your encoding
@Ash 2010-01-28 09:01:16
@Joel, Be careful with System.Text.Encoding.Default as it could be different on each machine it is run. That's why it's recommended to always specify an encoding, such as UTF-8.
@user541686 2012-04-30 07:55:05
You don't need the encodings unless you (or someone else) actually intend(s) to interpret the data, instead of treating it as a generic "block of bytes". For things like compression, encryption, etc., worrying about the encoding is meaningless. See my answer for a way to do this without worrying about the encoding. (I might have given a -1 for saying you need to worry about encodings when you don't, but I'm not feeling particularly mean today. :P)
@Roland 2013-03-26 14:49:50
Good discussion, sometimes I need one of the above alternatives. But also looks like: "One fool can ask more than seven wise men can answer" :-)
@chiccodoro 2013-07-17 07:57:28
+1; @Mehrdad: The
GetString method is and interpretation of the output of the
GetBytes method. This is why you have to worry to use the same encoding in both methods.
@Millie Smith 2016-02-05 23:39:01
I think it's important to note that it doesn't "[depend] on the encoding of your string". .NET hides this from you. From what I can tell, a String is represented by a sequence of System.Chars, which are represented as UTF-16. What matters is that you must store the bytes in some encoding and know to retrieve them with the same encoding. To not do that is the same as password-protecting your files and trying to use a different password to unprotect them.
@Nyerguds 2019-12-17 13:28:33
I don't think the encoding in .Net is actually UTF-16; that implies control bits. It simply saves all text as raw 16-bit code words, without the expandability. From what I've seen, this also means it doesn't support unicode above code word 0xFFFF
@Jodrell 2014-11-25 10:29:12
If you really want a copy of the underlying bytes of a string, you can use a function like the one that follows. However, you shouldn't please read on to find out why.
EntryPoint = "memcpy",
CallingConvention = CallingConvention.Cdecl,
SetLastError = false)]
private static extern unsafe void* UnsafeMemoryCopy(
void* destination,
void* source,
uint count);
public static byte[] GetUnderlyingBytes(string source)
var length = source.Length * sizeof(char);
var result = new byte[length];
fixed (char* firstSourceChar = source)
fixed (byte* firstDestination = result)
var firstSource = (byte*)firstSourceChar;
return result;
This function will get you a copy of the bytes underlying your string, pretty quickly. You'll get those bytes in whatever way they are encoding on your system. This encoding is almost certainly UTF-16LE but that is an implementation detail you shouldn't have to care about.
It would be safer, simpler and more reliable to just call,
In all likelihood this will give the same result, is easier to type, and the bytes will always round-trip with a call to
@Jarvis Stark 2014-09-09 11:30:51
You can use the following code for conversion between string and byte array.
string s = "Hello World";
// String to Byte[]
byte[] byte1 = System.Text.Encoding.Default.GetBytes(s);
byte[] byte2 = System.Text.ASCIIEncoding.Default.GetBytes(s);
// Byte[] to string
string str = System.Text.Encoding.UTF8.GetString(byte1);
@r.hamd 2015-09-09 13:19:57
VUPthis one solved my problem ( byte[] ff = ASCIIEncoding.ASCII.GetBytes(barcodetxt.Text);)
@Shyam sundar shah 2013-09-02 11:21:11
You can use following code to convert a
string to a
byte array in .NET
string s_unicode = "abcéabc";
byte[] utf8Bytes = System.Text.Encoding.UTF8.GetBytes(s_unicode);
@Avlin 2012-10-11 09:45:46
simple code with LINQ
string s = "abc"
byte[] b = s.Select(e => (byte)e).ToArray();
EDIT : as commented below, it is not a good way.
but you can still use it to understand LINQ with a more appropriate coding :
string s = "abc"
byte[] b = s.Cast<byte>().ToArray();
@WynandB 2013-10-25 04:36:21
It's hardly more faster, let alone most fastest. It's certainly an interesting alternative, but it's essentially the same as
Encoding.Default.GetBytes(s) which, by the way, is way faster. Quick testing suggests that
Encoding.Default.GetBytes(s) performs at least 79% faster. YMMV.
@Hans Kesting 2013-12-18 08:57:07
Try it with a
€. This code will not crash, but will return a wrong result (which is even worse). Try casting to a
short instead of
byte to see the difference.
@Tom Blodget 2013-12-02 04:43:48
This is a popular question. It is important to understand what the question author is asking, and that it is different from what is likely the most common need. To discourage misuse of the code where it is not needed, I've answered the later first.
Common Need
Every string has a character set and encoding. When you convert a
System.String object to an array of
System.Byte you still have a character set and encoding. For most usages, you'd know which character set and encoding you need and .NET makes it simple to "copy with conversion." Just choose the appropriate
Encoding class.
// using System.Text;
Encoding.UTF8.GetBytes(".NET String to byte array")
The conversion may need to handle cases where the target character set or encoding doesn't support a character that's in the source. You have some choices: exception, substitution or skipping. The default policy is to substitute a '?'.
// using System.Text;
var text = Encoding.ASCII.GetString(Encoding.ASCII.GetBytes("You win €100"));
// -> "You win ?100"
Clearly, conversions are not necessarily lossless!
Note: For
System.String the source character set is Unicode.
The only confusing thing is that .NET uses the name of a character set for the name of one particular encoding of that character set.
Encoding.Unicode should be called
That's it for most usages. If that's what you need, stop reading here. See the fun Joel Spolsky article if you don't understand what an encoding is.
Specific Need
Now, the question author asks, "Every string is stored as an array of bytes, right? Why can't I simply have those bytes?"
He doesn't want any conversion.
From the C# spec:
Character and string processing in C# uses Unicode encoding. The char type represents a UTF-16 code unit, and the string type represents a sequence of UTF-16 code units.
So, we know that if we ask for the null conversion (i.e., from UTF-16 to UTF-16), we'll get the desired result:
Encoding.Unicode.GetBytes(".NET String to byte array")
But to avoid the mention of encodings, we must do it another way. If an intermediate data type is acceptable, there is a conceptual shortcut for this:
".NET String to byte array".ToCharArray()
That doesn't get us the desired datatype but Mehrdad's answer shows how to convert this Char array to a Byte array using BlockCopy. However, this copies the string twice! And, it too explicitly uses encoding-specific code: the datatype
The only way to get to the actual bytes the String is stored in is to use a pointer. The
fixed statement allows taking the address of values. From the C# spec:
[For] an expression of type string, ... the initializer computes the address of the first character in the string.
To do so, the compiler writes code skip over the other parts of the string object with
RuntimeHelpers.OffsetToStringData. So, to get the raw bytes, just create a pointer to the string and copy the number of bytes needed.
// using System.Runtime.InteropServices
unsafe byte[] GetRawBytes(String s)
if (s == null) return null;
var codeunitCount = s.Length;
/* We know that String is a sequence of UTF-16 codeunits
and such codeunits are 2 bytes */
var byteCount = codeunitCount * 2;
var bytes = new byte[byteCount];
fixed(void* pRaw = s)
Marshal.Copy((IntPtr)pRaw, bytes, 0, byteCount);
return bytes;
As @CodesInChaos pointed out, the result depends on the endianness of the machine. But the question author is not concerned with that.
@Jan Hettich 2014-02-04 02:33:44
In general, is not correct to set
byteCount to twice the string length. For Unicode code points outside the Basic Multilingual Plane, there will be two 16-bit code units for each character.
@Tom Blodget 2014-02-04 02:35:29
@Jan That's correct but string length already gives the number of code-units (not codepoints).
@Jan Hettich 2014-02-04 05:42:50
Thanks for pointing that out! From MSDN: "The
Length property [of
String] returns the number of
Char objects in this instance, not the number of Unicode characters." Your example code is therefore correct as written.
@supercat 2014-11-12 22:29:24
I don't think
Char is really an "encoding-specific" type; from what I can tell, there is a specified 1:1 relationship between
Char values and
UInt16 values, any
Char[] can be converted to a string of the same length, and any such string may be converted to a
Char[] equal to the original, whether or not the sequence of
Char values ever formed a valid UTF-16 string.
@Tom Blodget 2014-11-13 00:15:04
@supercat "The char type represents a UTF-16 code unit, and the string type represents a sequence of UTF-16 code units."—_C# 5 Specification._ Although, yes, there is nothing that prevents a invalid Unicode string:
new String(new []{'\uD800', '\u0030'})
@supercat 2014-11-13 17:50:15
@TomBlodget: I can't find anything which indicates that all values 0x0000-0xFFFF may be regarded as "code units", but the term "sequence of code units" would imply that the type could accommodate sequences of code units which do not represent sequences of code points. I really don't know any type other than
String that better encapsulates the concept of "immutable sequence of 16-bit values"; because
System.String has special Runtime support which is not available for any other type, it can offer better performance for many operations than would be possible with any other type.
@supercat 2014-11-13 17:56:04
@TomBlodget: Interestingly, if one takes instances of
Globalization.SortKey, extracts the
KeyData, and packs the resulting bytes from each into a
String [two bytes per character, MSB first], calling
String.CompareOrdinal upon the resulting strings will be substantially faster than calling
SortKey.Compare on the instances of
SortKey, or even calling
memcmp on those instances. Given that, I wonder why
KeyData returns a
Byte[] rather than a
@vexe 2015-03-14 13:55:13
@TomBlodget +1 great answer! For the sake of completeness, it'd be nice to add how to back in reverse. This worked for me:
unsafe string GetString(byte[] bytes) { fixed (byte* bptr = bytes) { char* cptr = (char*)(bptr); var result = new string(cptr, 0, bytes.Length / 2); return result; } }
@Martin Capodici 2015-06-30 02:38:16
Alas, the right answer, but years too late, will never has as many votes as the accepted. Due to TL;DR people will think the accepted answer rocks. copyenpastit and up-vote it.
@Gerard ONeill 2015-08-18 15:59:37
Love this answer because of the approach, but it is wrong -- A surrogate pair would be a single code unit, but would be 4 bytes. So codeunitcount * 2 is not correct.
@Tom Blodget 2015-08-18 19:19:31
@GerardONeill Thanks for the feedback. According the the C# spec, a .NET string is counted sequence of UTF-16 code units. A codepoint is encoded in one or more code units. In the case of UTF-16, that's one or two. When two, they are the "high" surrogate followed by the "low" surrogate. So,
codeunitcount * 2 is the correct number of bytes for a code unit. The code does not count codepoints at all.
@Gerard ONeill 2015-08-18 19:47:21
Sorry, I didn't know the semantics of 'Code Unit'. Did not realize the horror of String.Length with surrogates; it seemed obvious that length would count full blown chars (codepoints). So yes, what you have here will work. This also explains why and how unmatched surrogates are allowed in strings.
@Tom Blodget 2015-08-19 23:59:38
@GerardONeill Yes, horror. I had been assuming that strings had to valid Unicode (including matching surrogates) but, alas, nothing says that it has to be true.
@user541686 2018-01-28 04:27:51
@TomBlodget: You don't need
fixed or
unsafe code, you can also do
var gch = GCHandle.Alloc("foo", GCHandleType.Pinned); var arr = new byte[sizeof(char) * ((string)gch.Target).Length]; Marshal.Copy(gch.AddrOfPinnedObject(), arr, 0, arr.Length); gch.Free();
@Tom Blodget 2018-01-28 19:05:33
@Mehrdad Yes, that is also a good answer that meets the rather limiting non-functional constraints of the question asker. I think Pinned and fixed amount to the same thing but it does eliminate the need for unsafe.
@Thomas Eding 2013-09-27 23:26:41
OP's question: "How do I convert a
string to a
byte array in .NET (C#)?" [sic]
You can use the following code:
static byte[] ConvertString (string s) {
return new byte[0];
As a benefit, encoding does not matter! Oh wait, this is an ecoding... it's just trivial and highly lossy.
@Lodewijk 2014-04-28 12:44:01
It's not a conversion. It's a new byte array. What the OP really needed was a pointer and memcpy. Or a cast: byte[] b = (byte[]) s;.
@TechNyquist 2014-10-14 07:18:05
Furthermore, "s" isn't even used here. Definitely not a solution.
Related Questions
Sponsored Content
14 Answered Questions
[SOLVED] How do I get a substring of a string in Python?
44 Answered Questions
[SOLVED] How do I create an Excel (.XLS and .XLSX) file in C# without installing Microsoft Office?
14 Answered Questions
[SOLVED] How do I turn a C# object into a JSON string in .NET?
- 2011-06-01 12:59:21
- Hui
- 1392407 View
- 960 Score
- 14 Answer
- Tags: c# .net json serialization
44 Answered Questions
[SOLVED] How do you convert a byte array to a hexadecimal string, and vice versa?
15 Answered Questions
[SOLVED] How to convert UTF-8 byte[] to string?
- 2009-06-16 18:47:00
- 1114016 View
- 943 Score
- 15 Answer
- Tags: c# .net arrays string type-conversion
27 Answered Questions
[SOLVED] Why not inherit from List<T>?
- 2014-02-11 03:01:36
- Superbest
- 184931 View
- 1416 Score
- 27 Answer
- Tags: c# .net list oop inheritance
16 Answered Questions
4 Answered Questions
[SOLVED] How do I convert struct System.Byte byte[] to a System.IO.Stream object in C#?
9 Answered Questions
[SOLVED] How do I encode and decode a base64 string?
5 Answered Questions
[SOLVED] \d is less efficient than [0-9]
- 2013-05-18 07:18:40
- weston
- 84070 View
- 1251 Score
- 5 Answer
- Tags: c# regex performance