linux日常运维命令

查文件删文件

查文件

#查文件
sudo find /opt -name '.DS_Store'
#查文件删文件
sudo find /opt -name '.DS_Store' -delete

进入乱码名称文件夹

cd `ls | awk 'NR==1'`

同步系统时钟

时间服务器：

time.nist.gov  
time.nuri.net  
0.asia.pool.ntp.org  
1.asia.pool.ntp.org  
2.asia.pool.ntp.org  
3.asia.pool.ntp.org

debian: apt-get install ntpdate  
centos: yum install ntpdate

sudo ntpdate s1b.time.edu.cn
sudo ntpdate 0.asia.pool.ntp.org 
sudo ntpdate time.nist.gov 
sudo date -s "26 OCT 2021 14:02:30"
sudo date -s "18 MAY 2022 15:39:30"

检查大文件&清理docker日志

2020-07-24 今天拷贝一个文件到服务器上提示空间不足，才发现机器空间已满了，难怪这阵子按tab键自动填充命令都会报错，使用以下命令查找大文件：

find / -type f -size +800M
find / -type f -print0 | xargs -0 du -h | sort -rh | head -n 10

结果返回了挺多docker的log文件，服务用的也不频繁，开发者究竟定时打了多少日志，竟然累积了如此多的量。

使用du命令查看日志文件的大小，-m表示单位为MB

du -m $log

定位到一个文件占了几十G的内存，使用rm -rf 方式删除日志后，通过df -h会发现磁盘空间并没有释放，原因：在Linux或者Unix系统中，通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的（有一个进程正在使用），那么进程将仍然可以读取该文件，磁盘空间也一直被占用。

正确姿势是cat /dev/null > $log，当然你也可以通过rm删除后重启docker

断点下载大模型

wget -c https://huggingface.co/stabilityai/stable-diffusion-2-1-unclip/resolve/main/sd21-unclip-l.ckpt

docker 日志大小限制

image: my-app:latest
    logging:
        driver: "json-file"
        options:
            max-file: 5
            max-size: 10m

解压文件

mkdir目标文件夹
-d 后紧接目标文件夹

 unzip -o -d ./v2ray ./v2ray-linux-64.zip
 
# tar解压文件
tar -zxvf abc.tgz
#分割后的压缩包解压命令如下
cat cm-11.tar.gz.* | tar -zxv

压缩文件

mkdir**目标文件夹
-d 后紧接目标文件夹

# 将文件 abc文件夹 进行压缩时，排除1.txt，压缩后的文件名为 abc.tgr
tar --exclude=abc/1.txt -zcvf abc.tgz abc

# 压缩后的文件太大，需要将cm-11.tar.gz分割成N个指定大小的文件，怎么办？一条命令搞定
split -b 4000M -d -a 1 cm-11.tar.gz cm-11.tar.gz.
#使用split命令，-b 4000M 表示设置每个分割包的大小，单位还是可以k
#-d "参数指定生成的分割包后缀为数字的形式
#-a x来设定序列的长度(默认值是2)，这里设定序列的长度为1

查看DNS

cat /etc/resolv.conf

linux内存占满卡死，结束docker进程

kill -SIGUSR1 $(cat /var/run/docker.pid)
kill -9 $(cat /var/run/docker.pid)
kill -5 $(cat /var/run/docker.pid)

根据进程名查看进程id

ps aux | grep 进程名(或者ps -ef | grep 进程名)

通过进程id查看占用的端口

netstat -nap | grep 进程id
netstat -ltnp

通过端口号查看占用的进程id

netstat -nap | grep 端口号

Display line numbers:

:set nu

Stop showing the line numbers:

:set nonu

防火墙

1.查看防火墙状态

systemctl status firewalld
此处如果提示firewalld.service could not be found.则说明你的系统没有装防火墙，则不需要设置。

2.开放指定端口

此处以开放1024端口为例：

firewall-cmd --zone=public --add-port=443/tcp --permanent
–permanent参数表示永久生效，没有此参数重启后失效。

3.重启防火墙

firewall-cmd --reload #重启firewall
systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动
firewall-cmd --state #查看默认防火墙状态（关闭后显示notrunning，开启后显示running）

下载url集合

#-c检查是否已下载
wget -c -i urls.txt

设置ssh免密码登陆linux

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub | ssh mike@10.211.55.4 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh mike@10.10.10.244 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys"

vim

# replace all occurrence string
:%s/foo/bar/g
# batch add prefix
Press Ctrl + v to enter into visual block mode.
Use ↑ / ↓ / j / k to select multiple lines.
Press Shift + i and start typing what you want.
After you press Esc, the text will be inserted into all the lines you selected.

batch change the symbolic links

cd ~/.cache/huggingface/hub/models--decapoda-research--llama-7b-hf/snapshots/5f98eefcc80e437ef68d457ad7bf167c2c6a1348

for link in config.json generation_config.json pytorch_model.bin.index.json special_tokens_map.json tokenizer.model tokenizer_config.json; do
    current_target=$(readlink "$link")
    new_target=${current_target/\/home\/mike\/.cache/\/home\/mike1\/.cache}
    echo "Changing target of $link from $current_target to $new_target"
    ln -sf "$new_target" "$link"
done


for i in {1..33}; do mv pytorch_model-000$i-of-00033.bin pytorch_model-$(printf "%05d" $i)-of-00033.bin; done

挂载硬盘

When you reboot the system, the temporary mounts under /media are not preserved. To automatically mount the filesystem when the system starts up, you’ll need to add an entry in the /etc/fstab file.

First, gather some information about the filesystem:

Determine the UUID (Universally Unique Identifier) of the filesystem using the blkid command:

sudo blkid

Look for the device with the matching mount point in the output and note its UUID.

Determine the filesystem type (e.g., ext4, ntfs, vfat, etc.). You can often find this information in the output of blkid or the mount command.

Next, create a permanent mount point for the filesystem, if it doesn’t already exist:

sudo mkdir -p /media/mike/8AC2C73DC2C72BEF

Then, edit the /etc/fstab file using your favorite text editor (e.g., nano, vi, or vim). In this example, we’ll use nano:

sudo nano /etc/fstab

Add an entry at the end of the file in the following format:

UUID=<filesystem_UUID> /media/mike/8AC2C73DC2C72BEF <filesystem_type> defaults 0 0

Replace <filesystem_UUID> with the UUID you obtained earlier, and <filesystem_type> with the appropriate filesystem type.

For example, if the UUID is 12345678-ABCD-1234-ABCD-1234567890AB and the filesystem type is ext4, the entry would look like this:

UUID=12345678-ABCD-1234-ABCD-1234567890AB /media/mike/8AC2C73DC2C72BEF ext4 defaults 0 0

Save the file and exit the text editor.

Finally, to mount the filesystem without rebooting, run:

sudo mount -a

This command mounts all filesystems listed in /etc/fstab. Now, your filesystem should be mounted at /media/mike/8AC2C73DC2C72BEF and will be automatically mounted at each system startup.

clean docker volume

Create a shell script called docker_volume_sizes.sh and add the following content:

#!/bin/bash

volumes=$(sudo docker volume ls -q)

echo "VOLUME NAME                   SIZE"
echo "-----------------------------------"

for volume in $volumes; do
  size=$(sudo docker run --rm -t -v "${volume}":/data alpine sh -c "du -sh /data 2>/dev/null | cut -f1")
  printf "%-27s %8s\n" "$volume" "$size"
done

This script will loop through all Docker volumes and use an Alpine Linux container to calculate the size of each volume.

Make the script executable:

chmod +x docker_volume_sizes.sh

Run the script:

./docker_volume_sizes.sh

This will output a table with Docker volume names and their sizes.

Note that running this script may take some time, as it creates and removes a temporary Alpine Linux container for each volume to calculate the size.

To determine which container is using a specific Docker volume, you can use the docker inspect command. For example, to find out which container is using the largest volume (in this case, the one with the ID f547aaf02bd3e2b2999a6f0bd0809dd99f85b1c95acacc19a244297eb04a08c7), run:

bashCopy code
sudo docker ps -a --format "\t\t" | grep "f547aaf02bd3e2b2999a6f0bd0809dd99f85b1c95acacc19a244297eb04a08c7"

This command lists all containers (including stopped ones) and their respective mount points, then filters the output to include only lines containing the volume ID you’re interested in.

Before deleting a volume, make sure to remove or stop any containers that are using it. You can stop a container using the following command (replace <container_id> with the actual container ID):

sudo docker stop <container_id>

And remove the container with:

sudo docker rm <container_id>

Note: Be cautious when deleting volumes and removing containers, as this process will permanently delete the associated data. Make sure to back up any important data before proceeding.

Once you have stopped and removed any containers using the volume, you can delete it using the docker volume rm command (replace <volume_id> with the actual volume ID):

sudo docker volume rm <volume_id>

In this case, to remove the largest volume:

sudo docker volume rm f547aaf02bd3e2b2999a6f0bd0809dd99f85b1c95acacc19a244297eb04a08c7

This will delete the volume and free up the disk space it was occupying.

You can list all dangling volumes using the following command:

sudo docker volume ls -f dangling=true

If you see the volume f547aaf02bd3e2b2999a6f0bd0809dd99f85b1c95acacc19a244297eb04a08c7 in the list of dangling volumes, it is safe to remove it, as it is not being used by any container.

To remove all dangling volumes and free up disk space, you can run:

sudo docker volume prune

日常命令