SevenDark 的博客漆夜之书

AI 笔记

2023-05-22T00:00:00+08:00

记录AI相关技术学习过程，包括一些python基础知识，算法介绍

配置MNIST数据转化

from torchvision import transforms

transformer = transforms.Compose([
    transforms.ToTensor()
])

通过transforms.Composse([...])方法可以自定义对MNIST数据的转化操作，可以添加例如转张量(tensor)，单一化（Normalization 将数据范围统一，防止数据差异过大）等操作

获取MNIST数据

from torchvision import datasets

data = datasets.MNIST('data', 
    download=True,
    train=False,
    transform=transformer)

从MNIST获取数据，'data'是数据存储目录

配置数据加载方式

from torch.utils.data import DataLoader

loader = DataLoader(data, batch_size=1, shuffle=True)

DataLoader是可迭代的内容，可使用foreach循环遍历内容，shuffle控制数据每次加载是否打乱顺寻，打乱顺序可避免对数据顺序的依赖

import itertools
import matplotlib.pyplot as plt

for image, label in itertools.isLice(loader, 5):
    plt.axis("off")
    plt.title(label.item())
    plt.imshow(image.squeeze(), cmap="gray")
    plt.show()

axis()方法控制是否展示图片xy标签
item()方法可获取标量值
squeeze()去除张量中size为1的维度
isLice()方法可以将迭代器截断，此处只取用前5个值

Python 定义模型前置知识

先放完整代码

import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28 * 28, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = torch.relu(fc1(x))
        x = torch.relu(fc2(x))
        x = fc3(x)
        return x

定义类

import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

    def forward():
        return 1

此处定义了一个最基本的模型类，(nn.Module)表示继承自nn.Module,def __init__() 定义了一个初始化方法，通过super()来调用父类方法

Linear 线性函数

import torch.nn as nn
import torch

l = nn.Linear(28 * 28, 128)
data = torch.randn((10, 28 * 28))
l(data)

Linear()函数第一个参数为输入张量大小，第二个为输出张量大小
torch.randn产生一个符合正态分布的张量，大小为(10, 28 * 28)
(5,)表示一个一维五个元素的张量
Linear()函数计算公式output = input @ weight^T + bias, weight(output_size, input_size) 初始为随机值，后续会根据损失函数进行调整, bias(output_size,) 偏置向量, @矩阵乘法, ^T矩阵转置

ReLu 函数

import torch

torch.relu()

激活函数，提供非线性变换，公式：ReLU(x) = max(0, x),取x大于0的区间

调整张量形状

x = x.view(x.size(0), -1)

view(a, b) 将x调整为形状为(a, b)的张量，参数-1表示自动计算维度大小
size()获取张量的形状，参数0表示获取第一个维度的大小

LangChain 基于本地txt的知识库问答

安装依赖

pip install langchain
pip install unstructured
pip install tabulate
pip install pdf2image
pip install chromadb
pip install tiktoken

整体代码示例

来源: https://www.youtube.com/watch?v=EnT-ZTrcPrg

from langchain import OpenAI
from langchain import VectorDBQA
from langchain.document_loaders import DirectoryLoader
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma


def qa_doc():
    loader = DirectoryLoader('docs', glob="**/*.txt")
    documents = loader.load()
    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
    texts = text_splitter.split_documents(documents)
    embeddings = OpenAIEmbeddings()
    docsearch = Chroma.from_documents(texts, embeddings)
    qa = VectorDBQA.from_chain_type(llm=OpenAI(), chain_type="stuff", vectorstore=docsearch)
    query = "What is Glueup"
    res = qa({"query": query})
    print(res["result"])
    print(res["source_documents"])


qa_doc()

https://playground.tensorflow.org/ 一个AI可视化的小工具

PyTorch Install

2023-05-19T00:00:00+08:00

Install Pytorch，And some AI tools

Install Pytorch

Install Anaconda

下载

https://www.anaconda.com/download/

教程

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#

# check envs
conda info --envs
conda create --name pytorch
conda activate pytorch

# check cuda version in powershell
nvidia-smi

Install PyTorch

https://pytorch.org/get-started/locally/#windows-installation

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

Install MNIST

conda install torchvision

Install matplotlib

该工具可以通过tensor中存储的二维数据绘制图片

conda install matplotlib

2022年的整年日记

2022-01-01T00:00:00+08:00

2023年来补下2022年的记录，没有学习什么内容，就草草写下2022年的日记吧，这一年都是在疫情中度过，疫情已经延续了很久，大家都已经备受折磨了，每天需要下楼做核酸，好在这些可怕的日子已经过去了，新冠疫情也被写进了历史，自己也走向了新的生活。

Docker Install on WSL2 Ubuntu20

2021-09-20T00:00:00+08:00

Install Docker on WSL2 Ubuntu20

Use tsinghua ubuntu mirrors

https://mirrors.tuna.tsinghua.edu.cn/help/ubuntu/

Install Docker with tsinghua mirrors

https://mirrors.tuna.tsinghua.edu.cn/help/docker-ce/

Install Docker desktop

https://hub.docker.com/editions/community/docker-ce-desktop-windows

WSL2

2021-09-20T00:00:00+08:00

WSL2 commands, restart wsl, set up wsl version

Install WSL2

Enable WSL

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

Enable VM feature

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

restart computer
download linux kernel

https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi

set wsl2 as default

wsl --set-default-version 2

check your distribution version

wsl --list --verbose

update distribution wsl version

wsl --set-version <distribution name> <versionNumber>

https://docs.microsoft.com/en-us/windows/wsl/install-win10

move WSL2 to another disk

Run powershell.exe as Administrator

PS C:\WINDOWS\system32> wsl -l
Windows Subsystem for Linux Distributions:
Ubuntu (Default)

# mkdir S:\ISOs\

PS C:\WINDOWS\system32> wsl --export Ubuntu S:\ISOs\ubuntu-wsl.tar

# mkdir w:\VMs

PS C:\WINDOWS\system32> cd w:\VMs
PS W:\VMs> mkdir ubuntu-wsl
PS W:\VMs> wsl --unregister Ubuntu
Unregistering...
PS W:\VMs> wsl --import Ubuntu W:\VMs\ubuntu-wsl S:\ISOs\ubuntu-wsl.tar
PS W:\VMs> wsl -l
Windows Subsystem for Linux Distributions:
Ubuntu (Default)

https://github.com/MicrosoftDocs/WSL/issues/412#issuecomment-575923176

Shutdown WSL

execute on cmd or powershell

wsl --shutdown

Github

2021-09-17T00:00:00+08:00

config the username and email, enable authorization cache, check your configs

config to cache your authorization info

config the username and email

git config --global user.name ""
git config --global user.email ""

config proxy

git config --global http.proxy "socks5://172.23.208.1:1080"
git config --global https.proxy "socks5://172.23.208.1:1080"

check your configs

git config -l

enable authorization cache

git config --global credential.helper cache

cleanup the authorization cache

git config --global --unset credential.helper

Kubernetes

2021-09-03T00:00:00+08:00

Install Kubernetes and run minikuber on your local, and deploy spring cloud on kubernetes.

How to install Docker on WSL2 https://blog.sevendark.cn/tec/Docker/

Install kubectl On Linux

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

kubectl reference https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands

Install Minikube

https://minikube.sigs.k8s.io/docs/start/

Enable Ingress

Enable ingress

minikube addons enable ingress

if enable failed, add proxy How to config ssh proxy https://blog.sevendark.cn/tec/SSH/#ssh-use-proxy

Open tunnel, Need keep this command alive

minikube tunnel

then make sure your ingress config bind on host: localhost, and access them by localhost

https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/index.md#docker-desktop

Kubernetes Compontents

Service

https://kubernetes.io/docs/concepts/services-networking/service/

Ingress

https://kubernetes.io/docs/concepts/services-networking/ingress/

Some command

check the logs

-n means namespace

kubectl logs --selector=app=zoomservice --tail 1 -n eb-services-qa

get the pod status

kubectl describe pod  zoomservice-5bd9b8d5bb-8xsbf -n eb-services-qa

get all things status in this namespace

kubectl get all --namespace=eb-services-qa

get ingress status

kubectl get ingress -n eb-services-qa

PHP Version Update

2020-11-27T00:00:00+08:00

Update PHP Version, Update Apache2 PHP Version, Swich PHP version on Ubuntu

Clean All PHP

sudo aptitude purge `dpkg -l | grep php| awk '{print $2}' |tr "\n" " "`

Update PHP version

Update php version

sudo apt update
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ondrej/php

install php 7.4

sudo apt update
sudo apt-get install -y php7.4

Swich PHP version

Swich Terminal PHP version

sudo update-alternatives --set php /usr/bin/php7.4

Swich php-fpm version ```sh

# Update Apache2 PHP version

- Install apache2 mods
```sh
sudo apt update
sudo apt-get install -y libapache2-mod-php7.4

Enter apache2 mods directory
```
cd /etc/apache2/mods-enabled
```
Check current php vesion
```
ls -l |grep php
```
Remove php[version].conf and php[version].load file

Create link file for new php version

ln -s ../mods-enabled/php[version].conf php[version].conf 
ln -s ../mods-enabled/php[version].load php[version].load 

Restart apache2
```
sudo systemctl restart apache2
```

https://websiteforstudents.com/install-php-8-0-on-ubuntu-20-04-18-04/

NextCloud Update

2020-09-24T00:00:00+08:00

Nextcloud Update And issues, “Step 4 is currently in process. Please reload this page later”,”a padding to disable MSIE and Chrome friendly error page”,”504 Gateway Time-out”

Common Update way

open settings

click Open Updater

Fix stuck on step 4

click Download now, just download .zip file use your computer.

use scp copy .zip file to your server /var/lib/nextcloud/data/updater-[random string]/downloads/.
update /var/lib/nextcloud/data/updater-[random string]/.step change start to end
Open Updater again, and click continue on top

Connection Refused Exception

e.g.

[Sat Mar 27 00:35:35.868963 2021] [php7:error] [pid 29625] [client 127.0.0.1:56664] PHP Fatal error:  Uncaught Doctrine\\DBAL\\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] Connection refused in /var/www/nextcloud/lib/private/DB/Connection.php:72

Try this:

sudo apt-get update
sudo apt-get install --reinstall apache2
sudo apt-get install libapache2-mod-authnz-external
sudo apt-get install libapache2-mod-security2

GitHub DNS (hosts)

2020-05-13T00:00:00+08:00

从去年年底开始加载Github头像，还有以raw方式打开github上的文件都会失败。添加DNS记录到hosts文件即可

windows hosts 文件位置：C:\Windows\System32\drivers\etc\hosts linux hosts 文件位置：/etc/hosts

82.113.4       github.com
82.112.4       gist.github.com
82.114.5       api.github.com
199.108.153    assets-cdn.github.com
232.68.133     raw.githubusercontent.com
232.68.133     gist.githubusercontent.com
232.68.133     cloud.githubusercontent.com
232.68.133     camo.githubusercontent.com
232.68.133     avatars0.githubusercontent.com
232.68.133     avatars1.githubusercontent.com
232.68.133     avatars2.githubusercontent.com
232.68.133     avatars3.githubusercontent.com
232.68.133     avatars4.githubusercontent.com
232.68.133     avatars5.githubusercontent.com
232.68.133     avatars6.githubusercontent.com
232.68.133     avatars7.githubusercontent.com
232.68.133     avatars8.githubusercontent.com

SevenDark 的博客 漆夜之书