Python学习笔记

python安装

创建桌面图标

sudo vim pycharm.desktop


[Desktop Entry]

Version=1.0

Type=Application

Name=Pycharm

Icon=/home/allen/Desktop/pycharm-2019.3.1/bin/pycharm.png

Exec=/home/allen/Desktop/pycharm-2019.3.1/bin/pycharm.sh

MimeType=application/x-py;

Name[en_US]=pycharm


右键勾选properties->permissions->allow exec file as system

软件列表显示

sudo cp pycharm.desktop /usr/share/applications/

Mac 下安装

卸载Python3.x

进入Applications删除相关程序包

假如还存在

使用

find / -name python3

来寻找系统中的Python文件

查找类似Python/framework/version的路径,进入删除即可


安装

brew install python3

centos7.4源码安装python3.7.4

tar -xvJf  Python-3.7.4.tar.xz
tar zvxf  Python-3.7.4.tgz

切换进入解压后的目录

cd Python-3.7.4
./configure --prefix=/usr/local/python37
make && make install

等待编译完成,如果报错需要按照下一个章节《常见编译错误》进行处理。
最后建立软连接,这样就可以直接敲命令了:

ln -s /usr/local/python37/bin/python3 /usr/bin/python3
ln -s /usr/local/python37/bin/pip3 /usr/bin/pip3

测试安装是否成功:

python3 --version
Python 3.7.4

mac 更换 pip 源

pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

如果是linux

mkdir ~/.pip

tee ~/.pip/pip.conf <<-'EOF'
[global]
index-url=http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com
EOF

如果是win

C:\Users\xx\pip\pip.ini

[global]
index-url=http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com

无需重启注销,直接生效

如果win下还是慢,注意xx-net之类设置成“取消全局代理”。把pip文件开头的source也改成这个

[[source]]
url = "https://pypi.tuna.tsinghua.edu.cn/simple"

Python启动web服务

python -m SimpleHTTPServer 80
python3 -m http.server 80

安装虚拟环境

venv python3已经包含在标准库中

## bash
# Mac/Linux
# You may need to run sudo apt-get install python3-venv first
python3 -m venv env

# Windows
py -3 -m venv env

安装pyenv

mac下安装

brew install pyenv
echo 'eval "$(pyenv init --path)"' >> ~/.z
pyenv install 3.8.5
pyenv local 3.8.5
pyenv global 3.8.5

安装

pip install virtualenv

使用

在当前目录下创建环境

virtualenv venv

若要使用系统的site-packages请添加参数:

virtualenv --system-site-packages venv

先安装需要指定版本的python

指定virtualenv中的python版本

virtualenv --no-site-packages --python=2.7 env

常见编译错误

错误1
zipimport.ZipImportError: can't decompress data; zlib not available

解决方法:

yum -y install zlib*

错误2
**ModuleNotFoundError: No module named '_ctypes'**

解决方法:

yum install -y libffi-devel

错误3

error: Microsoft Visual C++ 9.0 is required. Get it from http://aka.ms/vcpython27

错误4
win 更新pipAttributeError: 'NoneType' object has no attribute 'bytes'

# 方法1
python -m pip install -U pip
# 方法2
easy_install -U pip

错误5
win 下没有这个模块ModuleNotFoundError: No module named 'fcntl'

新建fncl.py文件

# C:\Python\Python36\Lib

错误6
fatal error: Python.h: No such file or directory

sudo yum install python-devel
def fcntl(fd, op, arg=0):
    return 0
 
def ioctl(fd, op, arg=0, mutable_flag=True):
    if mutable_flag:
        return 0
    else:
        return ""
 
def flock(fd, op):
    return
 
def lockf(fd, operation, length=0, start=0, whence=0):
    return

Windows不支持gunicorn

错误7
ImportError: cannot import name 'Feature' from 'setuptools'

setuptools版本问题, python3的setuptools版本太高

pip install --upgrade pip setuptools==45.2.0

卸载

brew uninstall python3

IDEA编写python

直接安装python插件即可

IDEA导入本地目录模块报错

打开Project Structure->Modules->选择源文件夹,点击Sources图标


brew 安装报/usr/local/bin目录不可写错误

sudo chown -R $(whoami) /usr/local/bin

Python头部信息的设置

File->settings->Editor->File and Code Templates->Files->Python Script

# -*- encoding:utf-8 -*-
"""
@author:Allen
@FileName:${NAME}.py
@Date:${DATE}  ${TIME}
@Description: 
"""


#[[$END$]]#

指定python解释器

/usr/bin/env 会自动去path中找python解释器,会使用path中第一个python解释器

#!/usr/bin/env python

Win 下 sublime Ctrl+B 运行 python 中文乱码

工具 -> 编译系统 -> 新编译系统

{  
    "cmd": ["python","-u","$file"],  
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",  
    "selector": "source.python",  
    "encoding": "cp936" 
}

用上面的代码替换掉"shell_cmd": "make", 保存当前配置文件为: "python_支持中文.sublime-build", 然后在工具->编译系统->选择 "python_支持中文"

python kafka

安装依赖

pip install kafka
pip install kafka-python

生产者

1 发送并忘记(不关注是否正常到达,不对返回结果做处理)

import pickle
import time
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=['ip:9092'],
                         key_serializer=lambda k: pickle.dumps(k),
                         value_serializer=lambda v: pickle.dumps(v))

start_time = time.time()

for i in range(0, 10000):
    print('------{}---------'.format(i))
    future = producer.send('test_topic', key='num', value=i, partition=0)

# 将缓冲区的全部消息push到broker当中
producer.flush()
producer.close()

end_time = time.time()
time_counts = end_time - start_time
print(time_counts)

2 同步发送(通过get方法等待Kafka的响应,判断消息是否发送成功)

import pickle
import time
from kafka import KafkaProducer
from kafka.errors import kafka_errors

producer = KafkaProducer(
    bootstrap_servers=['ip:9092'],
    key_serializer=lambda k: pickle.dumps(k),
    value_serializer=lambda v: pickle.dumps(v)
)

start_time = time.time()

for i in range(0, 10000):
    print('------{}---------'.format(i))
    future = producer.send(topic="test_topic", key="num", value=i)
    # 同步阻塞,通过调用get()方法进而保证一定程序是有序的.
    try:
        record_metadata = future.get(timeout=10)
        # print(record_metadata.topic)
        # print(record_metadata.partition)
        # print(record_metadata.offset)
    except kafka_errors as e:
        print(str(e))

end_time = time.time()
time_counts = end_time - start_time
print(time_counts)

3 异步发送+回调函数(消息以异步的方式发送,通过回调函数返回消息发送成功/失败)

import pickle
import time
from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers=['ip:9092'],
    key_serializer=lambda k: pickle.dumps(k),
    value_serializer=lambda v: pickle.dumps(v)
)

def on_send_success(*args, **kwargs):
    """
    发送成功的回调函数
    :param args:
    :param kwargs:
    :return:
    """
    return args


def on_send_error(*args, **kwargs):
    """
    发送失败的回调函数
    :param args:
    :param kwargs:
    :return:
    """
    return args


start_time = time.time()

for i in range(0, 10000):
    print('------{}---------'.format(i))
    # 如果成功,传进record_metadata,如果失败,传进Exception.
    producer.send(
        topic="test_topic", key="num", value=i
    ).add_callback(on_send_success).add_errback(on_send_error)

producer.flush()
producer.close()

end_time = time.time()
time_counts = end_time - start_time
print(time_counts)

除此之外,还能发送压缩数据流

def gzip_compress(msg_str):
    try:
        buf = StringIO.StringIO()
        with gzip.GzipFile(mode='wb', fileobj=buf) as f:
            f.write(msg_str)
        return buf.getvalue()
    except BaseException, e:
        print ("Gzip压缩错误" + e)


def gzip_uncompress(c_data):
    try:
        buf = StringIO.StringIO(c_data)
        with gzip.GzipFile(mode='rb', fileobj=buf) as f:
            return f.read()
    except BaseException, e:
        print ("Gzip解压错误" + e)


def send_kafka(topic_name, msg, key=None):
    if key is not None:
        producer = KafkaProducer(bootstrap_servers=["fdw8.fengjr.inc:9092","fdw9.fengjr.inc:9092","fdw10.fengjr.inc:9092"],
                                 key_serializer=gzip_compress, value_serializer=gzip_compress)
        r = producer.send(topic_name, value=msg, key=key)
    else:
        producer = KafkaProducer(bootstrap_servers=["fdw8.fengjr.inc:9092","fdw9.fengjr.inc:9092","fdw10.fengjr.inc:9092"],
                                 value_serializer=gzip_compress)
        r = producer.send(topic_name, value=msg)
    # producer.flush(timeout=5)
    producer.close(timeout=5)
    return r

python flask Gunicorn部署

安装python3

yum install python3
apt install python3
# 安装python开发依赖
sudo yum install python-devel
sudo apt install python3.6-dev
sudo apt install python3-venv

Ubuntu安装FFmpeg

apt install ffmpeg

报错

files list file for package 'libxmlsec1-nss:amd64' is missing final newline

编写n.py

#!/usr/bin/python

import os

dpkg_path = '/var/lib/dpkg/info/'
paths = os.listdir(dpkg_path)
for path in paths:
    path = dpkg_path + path
    f = open(path, 'a+')
    data = f.read()
    if len(data) > 1 and data[-1:] != '\n':
        f.write('\n')
        print 'added newline character to:', path
    f.close()  
sudo python n.py

centos7安装FFmpeg

https://www.vultr.com/docs/how-to-install-ffmpeg-on-centos

sudo yum install epel-release -y
sudo yum update -y
sudo shutdown -r now

sudo rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm

sudo yum install ffmpeg ffmpeg-devel -y
# 创建虚拟环境
cd /opt/py/vnc-services/
python3 -m venv venv
source venv/bin/activate

# 安装依赖
# sudo python3 setup.py install
python setup.py sdist
python setup.py build
python setup.py install
# flvscreen/flvscreen.c:489:27: error: storage size of ‘moduledef’ isn’t known
# make install
apt install ffmpeg


pip install -r requirements.txt

#export FLASK_APP=vncrecordingserver/t_api.py
#flask run

cd vncrecordingserver/
gunicorn app:app -c gunicorn.conf.py
# $(MODULE_NAME):$(VARIABLE_NAME) 模块名:变量名
gunicorn t_api:app -c gunicorn.conf.py

# 捕获控制台输出
gunicorn t_api:app -c gunicorn.conf.py --capture-output

gunicorn.conf.py

import multiprocessing
import os

# import gevent.monkey
# gevent.monkey.patch_all()

# debug = True
bind = "0.0.0.0:5000"
# chdir = '/opt/py/vnc-services/bin'   #gunicorn要切换到的目的工作目录
# loglevel = 'debug'
# pidfile = "/var/log/gunicorn/gunicorn.pid"
# accesslog = "/var/log/gunicorn/access.log"
# errorlog = "/var/log/gunicorn/debug.log"
daemon = True

# 启动的进程数
workers = os.getenv("WORKERS",1)
# workers = multiprocessing.cpu_count()
worker_class = 'gevent'
x_forwarded_for_header = 'X-FORWARDED-FOR'

gunicorn flask日志打印

在flask的app.py里配置

import logging

if __name__ != '__main__':
    gunicorn_logger = logging.getLogger('gunicorn.error')
    app.logger.handlers = gunicorn_logger.handlers
    app.logger.setLevel(gunicorn_logger.level)
最后由 不一样的少年 编辑于2021年12月03日 16:24