python convert b'123' to '123'

I'm using this code to get standard output from an external program:

>>> from subprocess import * >>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0] 

The communicate() method returns an array of bytes:

>>> command_stdout b'total 0\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2\n' 

However, I'd like to work with the output as a normal Python string. So that I could print it like this:

>>> print(command_stdout) -rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1 -rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2 

I thought that's what the binascii.b2a_qp() method is for, but when I tried it, I got the same byte array again:

>>> binascii.b2a_qp(command_stdout) b'total 0\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2\n' 

Does anybody know how to convert the bytes value back to string? I mean, using the "batteries" instead of doing it manually. And I'd like it to be ok with Python 3.

Replay

You need to decode the bytes object to produce a string:

>>> b"abcde"
b'abcde'

# utf-8 is used here because it is a very common encoding, but you
# need to use the encoding your data is actually in.
>>> b"abcde".decode("utf-8")
'abcde'

You need to decode the byte string and turn it in to a character (unicode) string.

b'hello'.decode(encoding)

or

str(b'hello', encoding)

I think this way is easy:

bytes = [112, 52, 52]
"".join(map(chr, bytes))
>> p44

I think what you actually want is this:

>>> from subprocess import *
>>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0]
>>> command_text = command_stdout.decode(encoding='windows-1252')

Aaron's answer was correct, except that you need to know WHICH encoding to use. And I believe that Windows uses 'windows-1252'. It will only matter if you have some unusual (non-ascii) characters in your content, but then it will make a difference.

By the way, the fact that it DOES matter is the reason that Python moved to using two different types for binary and text data: it can't convert magically between them because it doesn't know the encoding unless you tell it! The only way YOU would know is to read the Windows documentation (or read it here).

If you don't know the encoding, then to read binary input into string in Python 3 and Python 2 compatible way, use ancient MS-DOS cp437 encoding:

PY3K = sys.version_info >= (3, 0)

lines = []
for line in stream:
    if not PY3K:
        lines.append(line)
    else:
        lines.append(line.decode('cp437'))

Because encoding is unknown, expect non-English symbols to translate to characters of cp437 (English chars are not translated, because they match in most single byte encodings and UTF-8).

Decoding arbitrary binary input to UTF-8 is unsafe, because you may get this:

>>> b'\x00\x01\xffsd'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 2: invalid
start byte

The same applies to latin-1, which was popular (default?) for Python 2. See the missing points in Codepage Layout - it is where Python chokes with infamous ordinal not in range.

UPDATE 20150604: There are rumors that Python 3 has surrogateescape error strategy for encoding stuff into binary data without data loss and crashes, but it needs conversion tests [binary] -> [str] -> [binary] to validate both performance and reliability.

P.S. I used to be a Python fanboy like you, then I took an ordinal not in range.

Set universal_newlines to True, i.e.

command_stdout = Popen(['ls', '-l'], stdout=PIPE, universal_newlines=True).communicate()[0]

While @Aaron Maenpaa's answer just works, a user recently asked

Is there any more simply way? 'fhand.read().decode("ASCII")' [...] It's so long!

You can use

command_stdout.decode()

decode() has a standard argument

codecs.decode(obj, encoding='utf-8', errors='strict')

From http://docs.python.org/3/library/sys.html,

To write or read binary data from/to the standard streams, use the underlying binary buffer. For example, to write bytes to stdout, use sys.stdout.buffer.write(b'abc').

Category: python Time: 2009-03-03 Views: 1

Related post

  • Boost Python converter for std::tuple 2012-02-14

    I could not find a Boost Python converter which converts std::tuple, so I wrote one. This was tested with a g++ 4.7 snapshot on Debian squeeze. It uses C++ 11 features, specifically variadic templates, so requires a recent compiler. This is based on

  • I am trying to program a Raspberry PI 2 to replace an Ardiuno is it possible and is there a c++ to python converter online? [on hold] 2016-02-17

    I am trying to program a Raspberry PI 2 to replace a C++ program is it possible and is there a c++ to python converter online?

  • QGIS Python - Convert geometry from Postgres to QgsPoint() 2013-05-17

    I have made a query in Postgres to get some points to use with python. This points are in geometry data. How can I convert that geometry in decimals to use QgsPoint() in python? Thank you --------------Solutions------------- Very easy, you can use th

  • ArcGIS Python - Convert feature class names to lower case 2013-12-06

    Can anybody anyone point me to directions on how to convert file geodatabase feature class names to lower case by using python? Thanks --------------Solutions------------- Using a combination of string manipulation and renaming the feature classes us

  • Python converter: number-to-English - Project Euler 17 2014-01-13

    So I wrote this function to convert a given number to its interpretation in the English language as part of the Project Euler exercises. It works fine, but I sense that it's rather sloppy and inelegant, especially for Python where many things can be

  • QGIS Python - Convert the CRS of an in memory vector layer 2014-09-22

    In a python script of qgis I load layers from different datasources using the QgsVectorLayer() and the QgsRasterLayer() commands. The coordinate reference systems of the data sources differs. Since I have for some sources only the permission to read,

  • Python: Converting a string representing a time range to a struct_time object 2016-01-17

    You can parse a string representing a time, in Python, by using the strptime method. There are numerous working examples on stackoverflow: Converting string into datetime However, what if your string represented a time range, as opposed to a specific

  • Python converting a .dot into a .png [Error 2] The system cannot find the file specified 2016-01-19

    Relatively new to python. I'm trying to convert a .dot file into a .png using the following code: command = ["dot", "-Tpng", "C:\Users\Me\.spyder2\dt.dot", "-o", "dt.png"] subprocess.check_call(command) Bu

  • Python - Converting a string with escape characters to json 2016-01-19

    JSON objects are printed into my syslog file. I need to extract the string from the log and convert it into JSON. I don't have any problems extracting the string between '{' and '}', but certain strings have an escape character in them, and this is c

  • Qgis or Python: converting a CSV file of simple locations to raster? 2016-01-24

    I have a CSV file as follows: Diversity,Longitude,Latitude 7,114.99638889,-33.85333333 6,114.99790583,-33.85214594 10,115,-33.85416667 2,115.0252075,-33.84447519 I would like to convert it to a raster file with a set 'no data' value over most of the

  • Python - Convert cross sectional time-series data from quarterly to monthly frequency 2016-01-25

    Simplified Question: How do I merge monthly stock data from CRSP with quarterly accounting data from compustat using python (up sampling the quarterly data)? I am referring to the following question: Stackoverflow I have the same problem, however my

  • Python Convert Fraction Inside String to Integer 2016-02-10

    How can I convert an input entered as a fraction (eg, "4/2") into an integer in python 3.5? I have tried using both of the following codes: b = int(input("Please enter a value for b of the quadratic equation: ")) b = int(float(input(&q

  • Python: Convert RDF/XML to dictionnary 2016-02-16

    I am trying to parse an RDF/XML file into a Python dictionary in order to manipulate the data and then store the data in MongoDB. This is what the elements in my file look like: <rdf:Description rdf:about="http://bnb.data.bl.uk/id/resource/0068920

  • python - convert encoded json into utf-8 2016-03-10

    I have several json files that need to be handled in a python script, although it seems to NOT to be in a valid json format: 1) it has single-quotes instead of double-quotes 2) is encoded (see the u) 3) some letters are encoded e.g. \xf6 that represe

  • Python: converting tuple into 2D array 2016-01-17

    I want to convert tuple like t = [(4,10),(9,7),(11,2),(2,2)] into 2D array like: a = [[4,10],[9,7],[11,2],[2,2]] I tried a = [] for i in t: a.append(np.asarray(i)) print a is there any easier way? --------------Solutions------------- Use a list compr

  • Python: Convert non-specifically formatted string into dictionary? 2016-01-30

    I'm scraping my router's connected devices page to get a list of who's home. If I set a short DHCP release, I can do this pretty easily through a different method. However, I'd like to instead be able to turn the following string into a dictionary to

  • python convert scrape results to strings to count them 2016-01-31

    I need to count similar labels from html. I already got all of them in text format but to count them I suppose I need to: 1. Unite them 2. Convert to strings 3. count. Does anyone can help with it? from bs4 import BeautifulSoup from collections impor

  • Python convert string to valid JSON 2016-02-09

    I want to convert my own syntax (witch is like JSON) to the valid JSON string. Look at this string: GuiApplication { id: "myApp"; title: "My Application"; width: 100px; height: 100px; Button { id: "myBtn"; text: "This is

  • Python convert string to list (No .split()) 2016-02-12

    I've got a little problem: I have a String in my Database which is called actions. Now, I'm writing a method, which gets that string from the database (that works), and then I want to turn that string into a list. I know actions.split(), but this did

iOS development

Android development

Python development

JAVA development

Development language

PHP development

Ruby development

search

Front-end development

Database

development tools

Open Platform

Javascript development

.NET development

cloud computing

server

Copyright (C) avrocks.com, All Rights Reserved.

processed in 0.452 (s). 13 q(s)