FileChannel示例代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

public class FileChannelWriter {
    public static void main(String[] args) {
        String file = "test_write_using_filechannel.txt";
        try (RandomAccessFile writer = new RandomAccessFile(file, "rw");
             FileChannel channel = writer.getChannel()) {
            ByteBuffer buff = ByteBuffer.wrap("Hello".getBytes(StandardCharsets.UTF_8));

            channel.write(buff);
            System.out.println(channel.position());

            System.out.println(buff.position());
//            ByteBuffer buff2 = ByteBuffer.wrap("Hello world".getBytes(StandardCharsets.UTF_8));
            channel.position(3);
            buff.position(4);
            System.out.println(channel.position());
            channel.write(buff);
            System.out.println(channel.position());

            // verify
            RandomAccessFile reader = new RandomAccessFile(file, "r");
            System.out.println(reader.readLine());
            reader.close();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

发现channel写buffer的时候buffer自己的文件指针也随着变化 ，position如果是11，下次读写从12开始

https://medium.com/@xunnan.xu/its-all-about-buffers-zero-copy-mmap-and-java-nio-50f2a1bfc05c

linux sendfile是用来实现zero-copy的，就是java的FileChannel.transferTo
linux epoll是用来实现nio的selector的
前者是网络传输增强性能避免用户空间的buffer的，后者是io的模型用来减少所需要的线程数量

http://man7.org/linux/man-pages/man7/epoll.7.html
The epoll API performs a similar task to poll(2): monitoring multiple file descriptors to see if I/O is possible on any of them. The epoll API can be used either as an edge-triggered or a level-triggered interface and scales well to large numbers of watched file descriptors.

A new java.nio.channels.SelectorProvider implementation that is based on the Linux epoll event notification facility is included
If linux kernel >= 2.6 is detected, then the java.nio.channels.spi.SelectorProvider will use epoll.

1
2
3

public static Selector open() throws IOException {
    return SelectorProvider.provider().openSelector();
}

https://www.baeldung.com/java-nio-selector
https://github.com/mengxu2018/java-code-sample/tree/master/simple-nio/src/com/xu/nio/EchoTest.java

下载源码

cd kafka_source_dir
gradle

./gradlew jar （如果不能下载wrapper可以直接用local的gradle命令)
./gradlew idea

去掉编译warning（build.gradle)

tasks.withType(JavaCompile) {
options.encoding = ‘UTF-8’
增加 options.warnings = false

准备配置文件

把config下面的log4j.properties文件copy到kafka-2.3.0-src\core\src\main\resources\log4j.properties
同时修改config下面的server.properties的log.dirs=c:/tmp/kafka-logs

启动windows zookeeper

直接执行zkServer.cmd（提前修改zoo.cfg的路径为windows路径)

导入intellij

intellij open然后选择kafka-2.3.0-src\build.gradle

修改classpath

注意箭头的地方

运行main方法，注意-Dkafka.logs.dir=c:/tmp/log4j-kafka-logs是针对的log4j.properties,
和log.dirs=c:/tmp/kafka-logs里面的不一样

查错

java.io.IOException: No snapshot found, but there are log entries. Something，清空zoo.cfg所配置的dataDir目录下面的东西

https://github.com/mengxu2018/kafka-sample
例子在 producer/src/main/java/kafka/examples/xuhang/
包含一个producer，两个consumer group

今天遇到了新机器yum一直不成功，需要安装几个工具
sudo yum update
sudo yum install yum-utils
sudo yum groupinstall development

另外/etc/resolv.conf也有问题，可以加个nameserver 9.9.9.9

pip3.6 install torch torchvision

bin/kafka-server-start.sh -daemon config/server.properties
bin/kafka-server-stop.sh

broker.id不要写错

而且启动日志开头打印了[2019-08-03 04:36:30,484] INFO Client environment:host.name=218.93.250.18 (org.apache.zookeeper.ZooKeeper)很奇怪，这个是
try {
put(l, “host.name”,
InetAddress.getLocalHost().getCanonicalHostName());
} catch (UnknownHostException e) {
put(l, “host.name”, ““);
}
只有第一个机器正常
[2019-08-03 04:35:41,120] INFO Client environment:host.name=k8s-master (org.apache.zookeeper.ZooKeeper)

所以listeners=PLAINTEXT://192.168.77.132:9092 写ip才可以，hostname就不行

producer发送完成需要close

consumeroffsets何时创建?用户启动了一个消费者组(下称consumer group)进行消费或调用kafka-consumer-groups –describe等
https://www.cnblogs.com/huxi2b/p/8316289.html

https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

bin/kafka-topics.sh –create –bootstrap-server k8s-master:9092 –replication-factor 2 –partitions 6 –topic my-replicated-topic2
这个可以在任意一台集群机器执行，执行后可以看到每台机器都有4个 my-replicated-topic2-x的文件夹，有两个是replication，
每个replication都有一个leader

offset存储不一样了

Older versions of Kafka (pre 0.9) store offsets in ZK only, while newer version of Kafka, by default store offsets in an internal Kafka topic called __consumer_offsets (newer version might still commit to ZK though).

The advantage of committing offsets to the broker is, that the consumer does not depend on ZK and thus clients only need to talk to brokers which simplifies the overall architecture. Also, for large deployments with a lot of consumers, ZK can become a bottleneck while Kafka can handle this load easily (committing offsets is the same thing as writing to a topic and Kafka scales very well here – in fact, by default __consumer_offsets is created with 50 partitions IIRC).

What is advertised.listeners in Kafka?
https://medium.com/@iamsuraj/what-is-advertised-listeners-in-kafka-72e6fae7d68e
一般来说如果producer和broker都是再内网，则不需要advertised.listeners,如果需要外网访问broker，则需要advertised.listeners设置外网ip
https://www.cnblogs.com/justuntil/p/8037969.html

overview

Ever wondered why Java applications consume much more memory than the specified amount via the well-known -Xms and -Xmx tuning flags? For a variety of reasons and possible optimizations, the JVM may allocate extra native memory. These extra allocations can eventually raise the consumed memory beyond the -Xmx limitation.

In this tutorial we’re going to enumerate a few common sources of native memory allocations in the JVM, along with their sizing tuning flags, and then learn how to use Native Memory Tracking to monitor them.

Native Allocations

The heap usually is the largest consumer of memory in Java applications, but there are others. Besides the heap, the JVM allocates a fairly large chunk from the native memory to maintain its class metadata, application code, the code generated by JIT, internal data structures, etc. In the following sections, we’ll explore some of those allocations.

Metaspace

Similar to the Oracle JRockit and IBM JVM’s, the JDK 8 HotSpot JVM is now using native memory for the representation of class metadata, which is called Metaspace. This may have removed the old OOM Error.
In order to maintain some metadata about the loaded classes, The JVM uses a dedicated non-heap area called Metaspace.
Before Java 8, the equivalent was called PermGen or Permanent Generation. Metaspace or PermGen contains the metadata about the loaded classes rather than the instances of them, which are kept inside the heap.
The important thing here is that the heap sizing configurations won’t affect the Metaspace size since the Metaspace is an off-heap data area. In order to limit the Metaspace size, we use other tuning flags:

• -XX:MetaspaceSize and -XX:MaxMetaspaceSize to set the minimum and maximum Metaspace size
• Before Java 8, -XX:PermSize and -XX:MaxPermSize to set the minimum and maximum PermGen size

Threads

One of the most memory-consuming data areas in the JVM is the stack, created at the same time as each thread. The stack stores local variables and partial results, playing an important role in method invocations.

The default thread stack size is platform-dependent, but in most modern 64-bit operating systems, it’s around 1 MB. This size is configurable via the -Xss tuning flag.

In contrast with other data areas, the total memory allocated to stacks is practically unbounded when there is no limitation on the number of threads. It’s also worth mentioning that the JVM itself needs a few threads to perform its internal operations like GC or just-in-time compilations.

Code Cache

When the JVM compiles bytecode to assembly instructions(JIT compiler), it stores those instructions in a special non-heap data area called Code Cache. The code cache can be managed just like other data areas in the JVM. The -XX:InitialCodeCacheSize and -XX:ReservedCodeCacheSize tuning flags determine the initial and maximum possible size for the code cache.

Garbage Collection

The JVM is shipped with a handful of GC algorithms, each suitable for different use cases. All those GC algorithms share one common trait: they need to use some off-heap data structures to perform their tasks. These internal data structures consume more native memory.

Native Byte Buffers

The JVM is the usual suspect for a significant number of native allocations, but sometimes developers can directly allocate native memory, too. Most common approaches are the malloc call by JNI and NIO’s direct ByteBuffers.

refer

https://www.baeldung.com/native-memory-tracking-in-jvm
https://www.baeldung.com/jvm-parameters

“On a UNIX/linux system, everything is a file; if something is not a file, it is a process.”

主要讲述linux文件系统，分区，分区一个比较重要的作用就是安全，比如tomcat占据一个分区，自己本身的数据（比如视频文件)占据一个分区，自己本身的数据分区满了不会影响tomcat分区，不会影响系统分区

通常可以下面几个分区
a partition for user programs (/usr)
a partition containing the users’ personal data (/home)
a partition to store temporary data like print- and mail-queues (/var)
a partition for third party and extra software (/opt)

文件系统是针对操作系统而言的，对磁盘而言，就是物理结构一个个的扇区
So, if you have a storage device and you want to use a different file system on it, just copy the files off it first to back them up. Then, format that drive with a tool like Disk Management in Windows, GParted in Linux, or Disk Utility in Mac OS X and copy the backed up data to new filesystem.

• NTFS: Modern versions of Windows - since Windows XP - use the NTFS file system for their system partition.
• Ext2/Ext3/Ext4: You’ll often see the Ext2, Ext3, and Ext4 file systems on Linux. Ext2 is an older file systems, and it lacks important features like journaling - if the power goes out or a computer crashes while writing to an ext2 drive, data may be lost. Ext3 adds these robustness features at the cost of some speed. Ext4 is more modern and faster - it’s the default file system on most Linux distributions now, and is faster. Windows and Mac don’t support these file systems - you’ll need a third-party tool to access files on such file systems. For this reason, it’s often ideal to format your Linux system partitions as ext4 and leave removable devices formatted with FAT32 or NTFS if you need compatibility with other operating systems. Linux can read and write to both FAT32 or NTFS.

下面可以看到一块磁盘可以安装两个windows和linux，分别是ntfs文件系统和ext4，可以windows自带的磁盘管理分出一块空间出来，不需要格式化，然后u盘启动linux安装引导文件，把linux安装到空闲的分区，对linux来说可以自动识别ntfs的文件
https://tutorials.ubuntu.com/tutorial/tutorial-create-a-usb-stick-on-windows#0

https://www.serverbrain.org/administration-practice-2003/an-overview-of-disk-structure.html

https://medium.com/r/?url=http%3A%2F%2Ftldp.org%2FLDP%2Fintro-linux%2Fhtml%2Fsect_03_01.html
https://medium.com/r/?url=https%3A%2F%2Fwww.cs.uic.edu%2F~jbell%2FCourseNotes%2FOperatingSystems%2F11_FileSystemImplementation.html

安装第一台机器

三台机器192.168.77.130， 192.168.77.131， 192.168.77.132

三台机器都是如下的配置conf/zoo.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/root/zookeeper_/data
dataLogDir=/root/zookeeper_/log
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

server.1=192.168.77.130:2888:3888
server.2=192.168.77.131:2888:3888
server.3=192.168.77.132:2888:3888

每台机器都执行mkdir -p /root/zookeeper_/{logs,data}

三台机器按顺序执行下面的命令
echo “1” >> /root/zookeeper_/data/myid
echo “2” >> /root/zookeeper_/data/myid
echo “3” >> /root/zookeeper_/data/myid

启动

./zkServer.sh start

测试

[root@node2 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /root/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

bin/zkCli.sh -server 192.168.77.131:2181

可用性

A Zookeeper cluster is called an ensemble. Due to the algorithm used, it is recom‐ mended that ensembles contain an odd number of servers (e.g., 3, 5, etc.) as a major‐ ity of ensemble members (a quorum) must be working in order for Zookeeper to respond to requests. This means that in a three-node ensemble, you can run with one node missing. With a five-node ensemble, you can run with two nodes missing.

参考

dataLogDir : (No Java system property) This option will direct the machine to write the transaction log to the dataLogDir rather than the dataDir. This allows a dedicated log device to be used, and helps avoid competition between logging and snapshots.

查错

注意下载apache-zookeeper-3.5.5-bin.tar.gz而不是apache-zookeeper-3.5.5.tar.gz，否则会类找不到的错误